acceptodds
Under review as a conference paper at ICLR 2027

BLOC: Blockwise Composition for Adaptive-Rank Fine-Tuning in LLMs

Abstract

Parameter-efficient fine-tuning (PEFT) critically depends on how weight updates are parameterized with a small number of trainable parameters. A common PEFT approach assumes that the trainable matrices are formulated as a transformation of the product of two or more low-dimensional matrices. Such parameterizations often impose structural assumptions on the resulting update, such as favoring low- or high-rank solutions, rather than allowing the realized rank to emerge from optimization and vary adaptively across layers. To address this limitation, we propose BLOC, a PEFT method based on BLOckwise Composition. Specifically, BLOC partitions each weight update matrix into smaller blocks and constructs each block as a sparse linear combination of trainable block components defined directly in the block space. To control the number of trainable parameters, BLOC shares these components across compatible Transformer layers and uses a one-time gradient-guided calibration for task-informed initialization. Across three backbones and diverse reasoning tasks, BLOC achieves strong performance, while producing adaptive-rank updates and effectively reusing shared block components across layers.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Parameter-Efficient Fine-Tuning, High-Rank Adaptation, Cross-Layer Parameter Sharing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vqLbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BLOC composes weight-update blocks from a trainable component bank shared across layers. Task gradients determine the initial bank and fixed component selections; the components and their coefficients are then optimized. This provides a different parameter budget/rank trade-off from independent LoRA factors, while retaining exact merging.

Critical strengths
  1. The block-sharing design has support beyond its initialization scheme. Shared components permit distinct layer updates, and Appendix C favors global sharing at a fixed stated budget. The no-SVD variant scores 86.96 versus LoRA's 86.29, while full BLOC reaches 87.29 (Tables 1 and 4), suggesting that the parameterization itself accounts for much of the gain.

  2. The gains extend across models, tasks, and parameter budgets. Results cover three backbones with three-seed reporting and relevant HiRA/VB-LoRA comparisons. On Llama-2-7B, BLOC reaches 61.33 on GSM8K and 35.57 on HumanEval, versus 54.95 and 23.98 for the strongest listed competitors (Table 2). Figure 4 also shows an advantage at all four tested budgets.

Critical weaknesses
  1. The initialization ablation does not clearly isolate the benefit of gradient directions. Table 4 does not specify the no-SVD initialization precisely. Equations (7)–(9) imply unit-Frobenius-norm initial blocks for the SVD construction, so the alternative's initial scale needs to be reported or matched. Otherwise, the difference between full BLOC and the no-SVD variant may reflect scale as well as task-informed directions. This limits the interpretation of the initialization gain, while leaving the broader evidence for block sharing intact.

  2. The baseline comparison needs a clearer tuning protocol. Appendix A uses a learning rate of 1e-5 for LoRA/DoRA and 1e-3 for BLOC/HiRA without documenting the search ranges or selection rule. Different methods may need different rates, but the missing selection procedure makes it difficult to judge how much of the reported advantage comes from the method.

Questions for authors
  1. What search and selection procedure was used for each baseline, and what data formed the commonsense training mixture?
  2. How is the no-SVD variant initialized, and is its initial update scale comparable to full BLOC? An initial-versus-final spectrum comparison would also help qualify the adaptive-rank interpretation.
4 comments
AuthorsOpenReview ↗

Response by Authors, Part 1

W2. The baseline comparison needs a clearer tuning protocol. Appendix A uses a learning rate of 1e-5 for LoRA/DoRA and 1e-3 for BLOC/HiRA without documenting the search ranges or selection rule. Different methods may need different rates, but the missing selection procedure makes it difficult to judge how much of the reported advantage comes from the method. Q1. What search and selection procedure was used for each baseline, and what data formed the commonsense training mixture?

A1

We thank the reviewer for highlighting the need to clarify the hyperparameter tuning protocol and training data composition.

Learning-rate tuning. We conducted preliminary learning-rate comparisons over {1e-5, 1e-4, 1e-3} for LoRA, DoRA, and HiRA. For LoRA and DoRA, increasing the learning rate from 1e-5 to 1e-4 resulted in only minor improvements on commonsense reasoning tasks (approximately 0.1–0.3 percentage points), while substantially degrading performance on HumanEval. We therefore adopted 1e-5 as a unified learning rate across task families, prioritizing cross-task robustness rather than tuning independently for each benchmark. For HiRA, 1e-3 consistently achieved the best performance across all evaluated datasets. For VB-LoRA, we followed the original configuration (Li et al., 2024), using 1e-3 for the vector bank and 1e-2 for the routing logits.

Commonsense training mixture. We used the publicly available Commonsense170K dataset (Hu et al., 2023), following prior PEFT studies, including DoRA and HiRA. The dataset contains 170,420 instruction-formatted examples constructed from the training splits of eight commonsense reasoning benchmarks: BoolQ, PIQA, Social IQA, HellaSwag, WinoGrande, ARC-Easy, ARC-Challenge, and OpenBookQA. The original dataset converts task-specific training examples into a unified instruction-response format using predefined templates. We used this combined dataset for commonsense fine-tuning across all compared methods and evaluated each benchmark separately, reporting their macro-average accuracy.

We will clarify the learning-rate search ranges, selection rationale, and training data composition in the revised manuscript to improve reproducibility.

AuthorsOpenReview ↗

Response by Authors, Part 2

W1. The initialization ablation does not clearly isolate the benefit of gradient directions. Table 4 does not specify the no-SVD initialization precisely. Equations (7)–(9) imply unit-Frobenius-norm initial blocks for the SVD construction, so the alternative's initial scale needs to be reported or matched. Otherwise, the difference between full BLOC and the no-SVD variant may reflect scale as well as task-informed directions. This limits the interpretation of the initialization gain, while leaving the broader evidence for block sharing intact.

Q2. How is the no-SVD variant initialized, and is its initial update scale comparable to full BLOC? An initial-versus-final spectrum comparison would also help qualify the adaptive-rank interpretation.

A2: Initialization Ablation, Scale Matching, and Adaptive-Rank Behavior

We thank the reviewer for these constructive comments. We provide additional clarification and experimental evidence regarding the initialization procedure and spectral evolution of BLOC.

(1) Clarification of the No-SVD ablation.

The No-SVD ablation in Table 4 is designed to evaluate the effectiveness of BLOC's blockwise parameterization itself, rather than isolate the contribution of SVD. Specifically, we remove the entire task-informed initialization procedure, replacing it with uniform random initialization and random Top-K support assignment, while normalizing the initialized update blocks to unit Frobenius norm. All other parameterization and training settings remain unchanged.

The competitive performance without task-informed initialization demonstrates the effectiveness of BLOC's blockwise parameterization itself, while the improvement of Full BLOC highlights the additional benefit of task-informed initialization. We will rename this setting No Task-Informed Initialization and clarify its implementation in the revised manuscript.

(2) Initial-versus-final spectral comparison.

To further characterize the adaptive-rank behavior, we analyzed the evolution of the singular-value spectra of BLOC's reconstructed weight updates on Qwen3-4B, complementing the final-rank analysis on Llama-3-8B in the original manuscript.

ProjectionEffective Rank (Init. → Final)Top-10 Energy (Init. → Final)
WqW_q2.08 → 486.5599.02% → 13.18%
WkW_k2.17 → 408.3697.12% → 12.70%
WvW_v1.66 → 406.3898.90% → 15.85%
WupW_{\mathrm{up}}1.56 → 717.9997.79% → 12.13%
WdownW_{\mathrm{down}}2.86 → 680.2896.01% → 12.29%

Effective rank is computed from the entropy of the normalized squared singular values and averaged across layers. Top-10 Energy denotes the layer-averaged spectral energy captured by the ten largest singular values.

At initialization, the effective ranks range from 1.56 to 2.86, with over 96% of the spectral energy concentrated in the top ten singular directions. After fine-tuning, the effective ranks increase substantially to 406–718, while the top-ten energy concentration decreases to approximately 12–16%. Despite retaining the same blockwise parameterization throughout fine-tuning, BLOC develops distinct spectral structures across projection types, with FFN projections exhibiting higher effective ranks than attention projections.

These observations further support the adaptive-rank property of BLOC, as both the spectral dimensionality and the utilization of available rank capacity emerge adaptively through downstream optimization. We have included more detailed visualizations of the initial and final spectral distributions in Appendix B of the revised manuscript.

Thank you for the clarifications. Confirming that the random-initialization variant is also normalized to unit Frobenius norm resolves the scale-confound concern. Renaming it “No Task-Informed Initialization” is appropriate, since it removes both gradient-guided component construction and support assignment rather than isolating SVD alone. The spectral analysis is also useful, although the resulting ranks should be described as implicitly realized numerical ranks rather than explicitly selected ranks. Please also clarify BLOC’s own learning-rate search and selection procedure, which is not included in the response. With these revisions, I will keep my rating unchanged.

Thank you for your follow-up and constructive suggestions. We appreciate your acknowledgment of our clarifications regarding initialization and numerical rank, and we have incorporated the suggested terminology revisions into the manuscript.

Regarding BLOC's learning-rate search and selection procedure, we conducted a learning-rate comparison on Qwen3-4B. Specifically, we evaluated four candidate learning rates 3×10−33\times10^{-3}, 1×10−31\times10^{-3}, 5×10−45\times10^{-4}, and 1×10−41\times10^{-4}. Each configuration underwent the complete three-epoch training procedure, followed by evaluation on the eight commonsense reasoning benchmarks.

The results are summarized below.

Learning RateARC-CARC-EBoolQHellaSwagOBQAPIQASIQAWinoGAvg.
3e-388.5295.2070.5193.9088.4987.2880.2083.5785.96
1e-3 (Selected)88.4695.7572.2994.2390.5487.8880.0085.4386.82
5e-487.8794.7464.6890.8183.6084.4475.6175.6882.18
1e-486.8493.5965.0384.6078.0181.7571.0968.3178.65

We selected 1×10−31\times10^{-3} as the default learning rate based on the preliminary comparison. As shown above, this setting achieves the highest reported macro-average accuracy (86.82%) among the evaluated candidates, while 3×10−33\times10^{-3} also achieves competitive performance (85.96%). Smaller learning rates result in more noticeable performance degradation.

We have updated Appendix A.2 (Method-specific Configurations) of the revised manuscript to include the evaluated learning rates, the training and evaluation procedure, the selection criterion, and the corresponding results.

Thank you again for your valuable feedback.

Reviewer_EDfJOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper develops BLOC, a PEFT method that represents each adaptation block as a sparse composition of components selected from a shared global bank. The key idea is that sharing and recombining components across blocks yields higher adaptation capacity under a fixed parameter budget, leading to improved performance over standard low-rank adapters.

Critical strengths
  • The paper is well organized, and the proposed method and training pipeline are easy to follow.
  • Novel adapter parameterization based on shared components and sparse composition.
  • Promising empirical results across multiple models, tasks, and parameter budgets.
  • Extensive ablations studying initialization, component assignment, block size, and Top-K selection.
Critical weaknesses
  • BLOC implicitly imposes a Kronecker structural ΔWl=(Cl⊗I)Bstack\Delta W_l =(C_l \otimes I)B_{\mathrm{stack}}, where ClC_l is the corresponding row-sparse coefficient matrix. Thus, all update blocks must be constructed from the same dictionary, with one scalar coefficient applied to an entire component block. The paper does not justify when this restricted form is sufficiently expressive.
  • The paper only provides the rank-capacity bound, but gives no approximation guarantee or the required number of block components.
  • The method is closely related to block dictionary learning and sparse coding, but this connection and the corresponding theoretical literature are insufficiently discussed.
  • The “adaptive-rank” claim is somewhat limited because rank is not explicitly learned and controlled; it emerges within a capacity predetermined by SS and B\mathcal{B}.
Questions for authors
  • When is this shared block-dictionary parameterization ΔWl=(Cl⊗I)Bstack\Delta W_l =(C_l \otimes I)B_{\mathrm{stack}} sufficiently expressive, particularly when different layers require substantially different update directions? Can you provide approximation error bounds or theoretical guarantees on how the required number of components SS should scale with the desired update complexity?
  • How does BLOC relate to classical dictionary learning and sparse coding, and which results from that literature may apply to its expressiveness or optimization?
  • Since BLOC does not explicitly optimize rank, in what precise sense is it rank-adaptive rather than simply producing different post-training numerical ranks within the capacity fixed by SS and B\mathcal{B}?
6 comments
AuthorsOpenReview ↗

Response by Authors, Part 1

W1. BLOC implicitly imposes a Kronecker structural ΔWℓ=(Cℓ⊗Ib)Bstack\Delta W_\ell=(C_\ell\otimes I_b)B_{\mathrm{stack}}, where CℓC_\ell is the corresponding row-sparse coefficient matrix. Thus, all update blocks must be constructed from the same dictionary, with one scalar coefficient applied to an entire component block. The paper does not justify when this restricted form is sufficiently expressive.

Q1. When is this shared block-dictionary parameterization ΔWℓ=(Cℓ⊗Ib)Bstack\Delta W_\ell=(C_\ell\otimes I_b)B_{\mathrm{stack}} sufficiently expressive, particularly when different layers require substantially different update directions?

A1

We thank the reviewer for identifying this structural constraint. While BLOC restricts weight updates to a shared component space, it does not require different layers to have similar update directions.

Importantly, BLOC performs component sharing at the block level rather than the layer level. Even when different layers require substantially different overall weight updates, their local update blocks may still exhibit reusable structures. Moreover, the shared dictionary can accommodate both common and layer-specific update directions, as different blocks can select different subsets of components and learn independent composition coefficients. With SS shared components and KK selected components per block, BLOC permits up to (SK)\binom{S}{K} distinct local subspaces.

Our gradient-guided initialization further accounts for such cross-layer heterogeneity by constructing the shared dictionary from gradients across layers and independently assigning components to each block according to its own gradient direction.

Empirically, Figure 8 shows that different layers exhibit distinct component selection patterns despite sharing the same dictionary. Furthermore, Figure 5(a) shows that global sharing achieves the highest average accuracy (87.29%) among the tested sharing configurations under the same parameter budget. These results suggest that block-level component sharing can preserve sufficient layer-specific flexibility while enabling effective parameter reuse.

Nevertheless, when heterogeneous local update directions cannot be adequately covered by the shared dictionary, its finite capacity may limit expressivity. We further characterize this limitation through the approximation analysis below.

AuthorsOpenReview ↗

Response by Authors, Part 2

W2. The paper only provides the rank-capacity bound, but gives no approximation guarantee or the required number of block components.

Q2. Can you provide approximation error bounds or theoretical guarantees on how the required number of components S should scale with the desired update complexity?

A2

To characterize the representational capacity of BLOC, consider target update blocks across layers, vectorized and stacked into X∈RN×bdX\in\mathbb{R}^{N\times bd}. We define the optimal KK-sparse dictionary approximation error as

\quad \mathrm{s.t.}\ \|C_{i,:}\|_0\le K,$$ where $C\in\mathbb{R}^{N\times S}$ and $D\in\mathbb{R}^{S\times bd}$. The approximation error satisfies $$\sum_{j>S}\sigma_j^2(X) \le E_{S,K}^{*}(X) \le \sum_{j>S}\sigma_j^2(X)+\|A-A_K\|_F^2,$$ where $A=XV_S$ contains the projection coefficients onto the leading $S$ right singular directions, and $A_K$ retains the largest $K$ coefficients in absolute value per row. The lower bound follows from the rank constraint on $CD$, while the upper bound is obtained by constructing a feasible sparse approximation using the leading singular directions. The upper bound separates two sources of approximation error: finite dictionary capacity controlled by $S$ and coefficient sparsity controlled by $K$. To achieve a relative Frobenius approximation error of at most $\varepsilon$, a necessary condition on $S$ is $$S\ge\min\left\lbrace s:\frac{\sum_{j=1}^{s}\sigma_j^2(X)} {\sum_j\sigma_j^2(X)} \ge 1-\varepsilon^2 \right\rbrace.$$ Furthermore, if the target blocks lie in a union of $M$ linear subspaces, each of dimension at most $R$, exact representation is possible with a suitably constructed dictionary when $S\ge MR$ and $K\ge R$. These results indicate that the required dictionary size depends on the intrinsic dimensionality and diversity of local update directions across layers, rather than directly on the rank of individual weight-update matrices. Importantly, in practice, there is no predefined optimal update $X$ that BLOC must reconstruct, and different weight updates may achieve comparable downstream task performance. Therefore, accurately approximating an arbitrary target update is not a prerequisite for effective fine-tuning. Instead, BLOC needs sufficient representational capacity to support effective downstream task optimization. Empirically, Figure 4 shows that BLOC outperforms the evaluated baselines even with only five shared components per projection under the matched parameter budget. Increasing $S$ from 5 to 43 further improves average accuracy from 86.16% to 87.27%, demonstrating that BLOC achieves effective adaptation with a compact component bank while benefiting from additional representational capacity.
AuthorsOpenReview ↗

Response by Authors, Part 3

W3. The method is closely related to block dictionary learning and sparse coding, but this connection and the corresponding theoretical literature are insufficiently discussed.

Q3. How does BLOC relate to classical dictionary learning and sparse coding, and which results from that literature may apply to its expressiveness or optimization?

A3 BLOC is related to classical dictionary learning and sparse coding in its mathematical formulation, as update blocks are represented through sparse linear combinations of shared components. However, the fundamental distinction lies in the optimization objective. Classical reconstruction-based dictionary learning typically minimizes the reconstruction error of observed data:

min⁡C,D∥X−CD∥F2,\min_{C,D}\|X-CD\|_F^2,

whereas BLOC does not assume a predefined target weight update XX. Instead, it directly optimizes the shared components and block-specific coefficients under the downstream language-modeling objective:

min⁡C,DLLM(W0+ΔW(C,D)).\min_{C,D}\mathcal{L}_{\mathrm{LM}}(W_0+\Delta W(C,D)).

As shown in A2, theoretical insights from sparse dictionary approximation can be used to characterize the representational capacity of BLOC, particularly the relationship between dictionary size, composition sparsity, and approximation error. Nevertheless, such reconstruction-based analysis does not define the objective optimized by BLOC during fine-tuning.

From an optimization perspective, BLOC constructs task-informed components and determines sparse support assignments through gradient-guided initialization, followed by joint optimization of the shared components and block-specific coefficients via end-to-end backpropagation. Consequently, classical dictionary recovery and convergence guarantees do not directly extend to BLOC's optimization setting.

We will expand the Related Work section to clarify these connections and distinctions, including the applicability and limitations of existing theoretical results.

Reviewer_ityMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes BLOC, a parameter-efficient fine-tuning (PEFT) method that constructs weight updates through blockwise sparse linear composition. BLOC employs shared trainable components across Transformer layers, together with gradient-guided initialization and block-specific composition coefficients. Experiments demonstrate competitive performance against representative PEFT baselines.

Critical strengths

S1. BLOC provides an interesting alternative to conventional low-rank factorization, allowing more flexible weight updates under limited parameter budgets. S2. The combination of blockwise composition and cross-layer component sharing is well motivated and effectively supports parameter efficiency. S3. The paper presents strong experimental results across multiple models and tasks, supported by extensive ablation studies and analyses.

Critical weaknesses

W1. Eq. (2) can be rewritten as (\Delta W_\ell=(C_\ell\otimes I_b)B_{\mathrm{stack}}), implying that all layers sharing a component bank are restricted to a common subspace. Consequently, even if individual layers achieve high or heterogeneous ranks, their update directions cannot vary independently beyond this shared subspace. This introduces a trade-off between parameter sharing and layer-specific flexibility that is not sufficiently discussed in the paper. Clarifying this constraint would provide a more complete characterization of the proposed parameterization.

W2. According to Eqs. (8)–(9), the SVD-initialized components are orthonormal, while the composition coefficients are (\ell_2)-normalized. Consequently, every initialized update block has unit Frobenius norm, regardless of the magnitude of its corresponding task gradient. Although gradient magnitudes influence the construction of the shared components, they do not directly determine the initial update magnitude of individual blocks. Given that gradient magnitudes may vary substantially across layers and blocks, the rationale for imposing a uniform initialization scale is unclear.

W3. The core idea of BLOC, constructing weight updates through sparse combinations of shared components, is closely related to existing approaches. For instance, FrameFT [1] employs sparse coefficients over shared bases, while MoSA [2] explores non-low-rank adaptation through structured parameter sharing. Although BLOC introduces trainable block components and gradient-guided initialization, it remains unclear what distinctive representational or optimization advantages these designs provide over existing approaches. Therefore, the methodological novelty of BLOC could be better justified beyond the specific integration of existing ideas. [1] Adepu, H., et al. "Fine-Tuning of Transformer Models with Frames." ICML, 2026. [2] Wang, X., et al. "MoSA: Mosaic Shared Adaptation of Large Language Models." ICLR, 2026.

Questions for authors
  1. How does the shared subspace constraint affect layer-specific flexibility, and can rank-based metrics fully capture BLOC's expressivity?
  2. Why are all update blocks initialized with the same Frobenius norm, regardless of their gradient magnitudes?
  3. What are the key technical advantages of BLOC over recent shared-basis PEFT methods such as FrameFT and MoSA?
3 comments
AuthorsOpenReview ↗

Response by Authors, Part 1

W1&Q1. How does the shared subspace constraint affect layer-specific flexibility, and can rank-based metrics fully capture BLOC's expressivity?

A1 We thank the reviewer for this insightful comment. We further characterize this trade-off theoretically and empirically.

(1) Trade-off between update-space independence and capacity.

As the reviewer points out, ΔWℓ=(Cℓ⊗Ib)Bstack\Delta W_\ell=(C_\ell\otimes I_b)B_{\mathrm{stack}} constrains layer-wise updates to a common row space. However, sharing components does not require different layers to realize identical updates, as each block maintains its own support and independently trainable coefficients. Specifically, the first-order variation of Eq. (2) is

δ(ΔWℓ,i)=∑j∈Sℓ,iδcℓ,i,jBj+∑j∈Sℓ,icℓ,i,jδBj.\delta(\Delta W_{\ell,i})=\sum_{j\in\mathcal S_{\ell,i}}\delta c_{\ell,i,j}B_j+\sum_{j\in\mathcal S_{\ell,i}}c_{\ell,i,j}\delta B_j.

The first term enables independent local adaptation within the assigned subspace, whereas the second enables the subspace itself to evolve while coupling layers using the same components. Under a fixed total component budget SS across LL layers, sharing a bank among gg layers provides each layer access to approximately Sg/LSg/L components under uniform allocation. Although each block remains constrained to its selected components, a larger bank provides a richer set of candidate subspaces during support selection.

Moreover, at the parameterization level, a global bank can reproduce grouped banks by retaining their original support assignments, while allowing additional cross-group component selections, given the same total component budget and sparsity constraint. Thus, broader sharing trades the independence of component-level adaptation for more flexible allocation of representational capacity, while preserving independent block-specific composition.

(2) Quantifying the sharing–flexibility trade-off.

Let Gℓ,i=∇ΔWℓ,iLG_{\ell,i}=\nabla_{\Delta W_{\ell,i}}\mathcal L. By the chain rule,

\nabla_{B_j}\mathcal L=\sum_{\ell\in\mathcal G}H_{\ell,j},\quad H_{\ell,j}=\sum_{i:j\in\mathcal S_{\ell,i}}c_{\ell,i,j}G_{\ell,i},$$ where $\mathcal G$ denotes a group of layers sharing a component bank. While block-specific coefficients are optimized without direct cross-layer gradient aggregation, shared components aggregate gradient contributions from multiple layers. To quantify the resulting coupling, let $$U_{\ell,i\leftarrow m}=-\eta_B\sum_{j\in\mathcal S_{\ell,i}}c_{\ell,i,j}H_{m,j}$$ denote the first-order component-induced variation in block $(\ell,i)$ originating from layer $m$ under SGD. We define the **Cross-layer Leakage Ratio (CLR)** and **Gradient Aggregation Ratio (GAR)** as $$\mathrm{CLR}_ {\ell}= \frac{\sum_{m\in\mathcal G_{\ell}\setminus\\{\ell\\}}\lVert U_{m\leftarrow\ell}\rVert_F^2} {\sum_{m\in\mathcal G_{\ell}}\lVert U_{m\leftarrow\ell}\rVert_F^2}, \mathrm{GAR}= \frac{\sum_{\mathcal G,j}\lVert\sum_{\ell\in\mathcal G}H_{\ell,j}\rVert_F^2} {\sum_{\mathcal G,j}\sum_{\ell\in\mathcal G}\lVert H_{\ell,j}\rVert_F^2}.$$ Here, $U_{m\leftarrow\ell}$ is assembled from the corresponding block-level variations. CLR quantifies the fraction of squared component-induced variation propagated to other layers, with a uniform-influence reference of $(g-1)/g$ for $g$ equally affected layers. GAR characterizes the net gradient interactions induced by sharing, with values above or below one indicating reinforcement or cancellation, respectively. We analyze potential first-order update interactions using the pretrained model weights before fine-tuning and report the results alongside our sharing-granularity ablation (Figure 5(a) and Table 8). Layersperbank|Banksize|CLR(Uniform)|CLR(BLOC)|GAR|Avg.Acc.(%) -|-|-|-|-|- 1|1–2|0.000|0.000|1.000|86.695 2|2–3|0.500|0.332|1.068|86.525 4|5–6|0.750|0.581|1.207|86.911 8|10–11|0.875|0.746|1.420|86.826 32|43|0.969|0.903|2.016|**87.290** CLR increases with sharing scope but remains below the uniform-influence reference in all multi-layer settings, indicating stronger cross-layer coupling while retaining layer specificity. GAR above one indicates overall positive gradient interactions before fine-tuning. Under a fixed component budget, global sharing achieves the best accuracy (Figure 5(a) and Table 8), suggesting that greater accessible component capacity and cross-layer gradient aggregation can outweigh reduced layer-specific independence. Rank-based metrics and structural analysis characterize complementary aspects of BLOC's expressivity, capturing the spectral dimensionality of realized updates and the capacity and independence of admissible update spaces, respectively. CLR and GAR quantify cross-layer optimization interactions. These analyses clarify how BLOC balances parameter sharing and layer-specific flexibility. We have incorporated these discussions into Appendix C of the revised manuscript.
AuthorsOpenReview ↗

Response by Authors, Part 2

W2. According to Eqs. (8)–(9), the SVD-initialized components are orthonormal, while the composition coefficients are (\ell_2)-normalized. Consequently, every initialized update block has unit Frobenius norm, regardless of the magnitude of its corresponding task gradient. Although gradient magnitudes influence the construction of the shared components, they do not directly determine the initial update magnitude of individual blocks. Given that gradient magnitudes may vary substantially across layers and blocks, the rationale for imposing a uniform initialization scale is unclear.

Q2. Why are all update blocks initialized with the same Frobenius norm, regardless of their gradient magnitudes?

A2

Gradient magnitude is already incorporated into the initialization of the shared components through Eq. (6), where β = 0.5 partially preserves gradient magnitudes while mitigating the dominance of large-gradient blocks.

However, the gradient magnitude estimated from a few calibration batches does not necessarily reflect the update magnitude required throughout fine-tuning. We therefore adopt unit-norm initialization to provide a consistent initial scale across blocks while retaining task-informed update directions. Since both the shared components and block-specific coefficients are trainable, unit norm is only an initialization condition, rather than a constraint on the learned update magnitudes.

To empirically examine this design choice, we additionally scale the initial composition coefficients of each block proportionally to its gradient norm raised to the power γ, with RMS normalization across blocks within each projection to preserve the overall initialization energy. We keep the SVD components and Top-K selections unchanged. Here, γ = 0 corresponds to our original unit-norm initialization, while γ = 0.5 and γ = 1.0 progressively introduce gradient-dependent update magnitudes.

InitializationAvg. accuracy (%)
γ = 0 (BLOC)86.82
γ = 0.586.69
γ = 1.086.67

On Qwen3-4B across eight commonsense reasoning benchmarks, all three configurations achieve comparable average accuracy, with differences within 0.15 percentage points. These results suggest that explicitly incorporating gradient magnitudes into the initial block scales provides no additional benefit in the evaluated setting, supporting unit-norm initialization as a simpler yet equally effective choice.

AuthorsOpenReview ↗

Response by Authors, Part 3

W3. The core idea of BLOC, constructing weight updates through sparse combinations of shared components, is closely related to existing approaches. For instance, FrameFT [1] employs sparse coefficients over shared bases, while MoSA [2] explores non-low-rank adaptation through structured parameter sharing. Although BLOC introduces trainable block components and gradient-guided initialization, it remains unclear what distinctive representational or optimization advantages these designs provide over existing approaches. Therefore, the methodological novelty of BLOC could be better justified beyond the specific integration of existing ideas. [1] Adepu, H., et al. "Fine-Tuning of Transformer Models with Frames." ICML, 2026. [2] Wang, X., et al. "MoSA: Mosaic Shared Adaptation of Large Language Models." ICLR, 2026.

What are the key technical advantages of BLOC over recent shared-basis PEFT methods such as FrameFT and MoSA?

A3

While all three methods exploit parameter sharing, they differ in what is shared and how the shared representations are utilized and optimized. Their parameterizations are:

FrameFT (fixed shared bases):

=\frac{\alpha}{\sqrt{mn}}P_mC_\ell P_n^\top,$$ **MoSA (fine-grained scalar sharing):** $$\Delta W_\ell^{\mathrm{MoSA}} =\sum_{g=1}^{G_\ell}\theta_{\ell,g}M_{\ell,g},$$ **BLOC (learnable shared block components):** $$\Delta W_{\ell,i}^{\mathrm{BLOC}} =\sum_{j\in\mathcal S_{\ell,i}}c_{\ell,i,j}B_j.$$ These formulations reveal two distinctive technical advantages of BLOC: **(1) Task-adaptive shared representations.** FrameFT achieves parameter efficiency by optimizing sparse coefficients over fixed Fusion Frames. Increasing the number of trainable coefficients enriches the composition within the predefined representation. In contrast, BLOC jointly optimizes the shared matrix-valued components $B_j$ and block-specific coefficients $c_{\ell,i,j}$. This bilinear parameterization allows BLOC to adapt not only the local compositions but also the underlying shared representation itself. Increasing the component-bank size further expands the capacity for learning reusable, task-specific update patterns. **(2) Structured parameter reuse with local flexibility.** MoSA achieves high-rank adaptation by tying weight entries to shared scalars according to predefined group assignments. BLOC instead shares matrix-valued components across Transformer layers while retaining independent sparse composition coefficients for each update block. This allows different blocks to reuse common update patterns without requiring identical update values or compositions. Furthermore, BLOC's gradient-guided initialization identifies task-relevant shared components and assigns them to individual blocks according to their local gradient directions, providing a task-informed starting point for both shared representation learning and local adaptation. **Additional experiments.** To examine the practical benefits of these design differences, we evaluated FrameFT and MoSA on Qwen3-4B across eight commonsense reasoning benchmarks. We extended FrameFT to the LoRA-r16-equivalent budget by setting $\rho_m=\rho_n=64$ and $n_c=16(d_{\mathrm{in}}+d_{\mathrm{out}})$ per projection, retaining its fixed-frame sparse-coefficient formulation. | Method | BoolQ | PIQA | SIQA | ARC-C | ARC-E | OBQA | HellaS | WinoG | Avg. | |---|---:|---:|---:|---:|---:|---:|---:|---:|---:| | FrameFT | 71.58 | 87.61 | 79.08 | 89.88 | 95.50 | 87.70 | 94.26 | 83.57 | 86.15 | | MoSA | 68.06 | 87.93 | 78.86 | 88.67 | 95.67 | 89.58 | 93.85 | 82.79 | 85.68 | | BLOC | 72.29 | 87.88 | 80.00 | 88.46 | 95.75 | 90.54 | 94.23 | 85.43 | **86.82** | BLOC outperforms FrameFT and MoSA by **0.67 and 1.14 percentage points**, respectively. Therefore, the methodological contribution of BLOC lies not simply in combining sparse composition with parameter sharing, but in introducing a task-adaptive blockwise parameterization that jointly learns reusable matrix-valued update patterns and their local compositions. This provides a distinct mechanism for balancing shared representational capacity and layer-specific flexibility under a limited parameter budget. The additional results support the practical value of this parameterization. We will incorporate these comparisons and discussions into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.