- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
BLOC composes weight-update blocks from a trainable component bank shared across layers. Task gradients determine the initial bank and fixed component selections; the components and their coefficients are then optimized. This provides a different parameter budget/rank trade-off from independent LoRA factors, while retaining exact merging.
Critical strengths
-
The block-sharing design has support beyond its initialization scheme. Shared components permit distinct layer updates, and Appendix C favors global sharing at a fixed stated budget. The no-SVD variant scores 86.96 versus LoRA's 86.29, while full BLOC reaches 87.29 (Tables 1 and 4), suggesting that the parameterization itself accounts for much of the gain.
-
The gains extend across models, tasks, and parameter budgets. Results cover three backbones with three-seed reporting and relevant HiRA/VB-LoRA comparisons. On Llama-2-7B, BLOC reaches 61.33 on GSM8K and 35.57 on HumanEval, versus 54.95 and 23.98 for the strongest listed competitors (Table 2). Figure 4 also shows an advantage at all four tested budgets.
Critical weaknesses
-
The initialization ablation does not clearly isolate the benefit of gradient directions. Table 4 does not specify the no-SVD initialization precisely. Equations (7)–(9) imply unit-Frobenius-norm initial blocks for the SVD construction, so the alternative's initial scale needs to be reported or matched. Otherwise, the difference between full BLOC and the no-SVD variant may reflect scale as well as task-informed directions. This limits the interpretation of the initialization gain, while leaving the broader evidence for block sharing intact.
-
The baseline comparison needs a clearer tuning protocol. Appendix A uses a learning rate of 1e-5 for LoRA/DoRA and 1e-3 for BLOC/HiRA without documenting the search ranges or selection rule. Different methods may need different rates, but the missing selection procedure makes it difficult to judge how much of the reported advantage comes from the method.
Questions for authors
- What search and selection procedure was used for each baseline, and what data formed the commonsense training mixture?
- How is the no-SVD variant initialized, and is its initial update scale comparable to full BLOC? An initial-versus-final spectrum comparison would also help qualify the adaptive-rank interpretation.
4 comments
Response by Authors, Part 2
W1. The initialization ablation does not clearly isolate the benefit of gradient directions. Table 4 does not specify the no-SVD initialization precisely. Equations (7)–(9) imply unit-Frobenius-norm initial blocks for the SVD construction, so the alternative's initial scale needs to be reported or matched. Otherwise, the difference between full BLOC and the no-SVD variant may reflect scale as well as task-informed directions. This limits the interpretation of the initialization gain, while leaving the broader evidence for block sharing intact.
Q2. How is the no-SVD variant initialized, and is its initial update scale comparable to full BLOC? An initial-versus-final spectrum comparison would also help qualify the adaptive-rank interpretation.
A2: Initialization Ablation, Scale Matching, and Adaptive-Rank Behavior
We thank the reviewer for these constructive comments. We provide additional clarification and experimental evidence regarding the initialization procedure and spectral evolution of BLOC.
(1) Clarification of the No-SVD ablation.
The No-SVD ablation in Table 4 is designed to evaluate the effectiveness of BLOC's blockwise parameterization itself, rather than isolate the contribution of SVD. Specifically, we remove the entire task-informed initialization procedure, replacing it with uniform random initialization and random Top-K support assignment, while normalizing the initialized update blocks to unit Frobenius norm. All other parameterization and training settings remain unchanged.
The competitive performance without task-informed initialization demonstrates the effectiveness of BLOC's blockwise parameterization itself, while the improvement of Full BLOC highlights the additional benefit of task-informed initialization. We will rename this setting No Task-Informed Initialization and clarify its implementation in the revised manuscript.
(2) Initial-versus-final spectral comparison.
To further characterize the adaptive-rank behavior, we analyzed the evolution of the singular-value spectra of BLOC's reconstructed weight updates on Qwen3-4B, complementing the final-rank analysis on Llama-3-8B in the original manuscript.
| Projection | Effective Rank (Init. → Final) | Top-10 Energy (Init. → Final) |
|---|---|---|
| 2.08 → 486.55 | 99.02% → 13.18% | |
| 2.17 → 408.36 | 97.12% → 12.70% | |
| 1.66 → 406.38 | 98.90% → 15.85% | |
| 1.56 → 717.99 | 97.79% → 12.13% | |
| 2.86 → 680.28 | 96.01% → 12.29% |
Effective rank is computed from the entropy of the normalized squared singular values and averaged across layers. Top-10 Energy denotes the layer-averaged spectral energy captured by the ten largest singular values.
At initialization, the effective ranks range from 1.56 to 2.86, with over 96% of the spectral energy concentrated in the top ten singular directions. After fine-tuning, the effective ranks increase substantially to 406–718, while the top-ten energy concentration decreases to approximately 12–16%. Despite retaining the same blockwise parameterization throughout fine-tuning, BLOC develops distinct spectral structures across projection types, with FFN projections exhibiting higher effective ranks than attention projections.
These observations further support the adaptive-rank property of BLOC, as both the spectral dimensionality and the utilization of available rank capacity emerge adaptively through downstream optimization. We have included more detailed visualizations of the initial and final spectral distributions in Appendix B of the revised manuscript.
Thank you for the clarifications. Confirming that the random-initialization variant is also normalized to unit Frobenius norm resolves the scale-confound concern. Renaming it “No Task-Informed Initialization” is appropriate, since it removes both gradient-guided component construction and support assignment rather than isolating SVD alone. The spectral analysis is also useful, although the resulting ranks should be described as implicitly realized numerical ranks rather than explicitly selected ranks. Please also clarify BLOC’s own learning-rate search and selection procedure, which is not included in the response. With these revisions, I will keep my rating unchanged.
Thank you for your follow-up and constructive suggestions. We appreciate your acknowledgment of our clarifications regarding initialization and numerical rank, and we have incorporated the suggested terminology revisions into the manuscript.
Regarding BLOC's learning-rate search and selection procedure, we conducted a learning-rate comparison on Qwen3-4B. Specifically, we evaluated four candidate learning rates , , , and . Each configuration underwent the complete three-epoch training procedure, followed by evaluation on the eight commonsense reasoning benchmarks.
The results are summarized below.
| Learning Rate | ARC-C | ARC-E | BoolQ | HellaSwag | OBQA | PIQA | SIQA | WinoG | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| 3e-3 | 88.52 | 95.20 | 70.51 | 93.90 | 88.49 | 87.28 | 80.20 | 83.57 | 85.96 |
| 1e-3 (Selected) | 88.46 | 95.75 | 72.29 | 94.23 | 90.54 | 87.88 | 80.00 | 85.43 | 86.82 |
| 5e-4 | 87.87 | 94.74 | 64.68 | 90.81 | 83.60 | 84.44 | 75.61 | 75.68 | 82.18 |
| 1e-4 | 86.84 | 93.59 | 65.03 | 84.60 | 78.01 | 81.75 | 71.09 | 68.31 | 78.65 |
We selected as the default learning rate based on the preliminary comparison. As shown above, this setting achieves the highest reported macro-average accuracy (86.82%) among the evaluated candidates, while also achieves competitive performance (85.96%). Smaller learning rates result in more noticeable performance degradation.
We have updated Appendix A.2 (Method-specific Configurations) of the revised manuscript to include the evaluated learning rates, the training and evaluation procedure, the selection criterion, and the corresponding results.
Thank you again for your valuable feedback.
Response by Authors, Part 1
A1
We thank the reviewer for highlighting the need to clarify the hyperparameter tuning protocol and training data composition.
Learning-rate tuning. We conducted preliminary learning-rate comparisons over {1e-5, 1e-4, 1e-3} for LoRA, DoRA, and HiRA. For LoRA and DoRA, increasing the learning rate from 1e-5 to 1e-4 resulted in only minor improvements on commonsense reasoning tasks (approximately 0.1–0.3 percentage points), while substantially degrading performance on HumanEval. We therefore adopted 1e-5 as a unified learning rate across task families, prioritizing cross-task robustness rather than tuning independently for each benchmark. For HiRA, 1e-3 consistently achieved the best performance across all evaluated datasets. For VB-LoRA, we followed the original configuration (Li et al., 2024), using 1e-3 for the vector bank and 1e-2 for the routing logits.
Commonsense training mixture. We used the publicly available Commonsense170K dataset (Hu et al., 2023), following prior PEFT studies, including DoRA and HiRA. The dataset contains 170,420 instruction-formatted examples constructed from the training splits of eight commonsense reasoning benchmarks: BoolQ, PIQA, Social IQA, HellaSwag, WinoGrande, ARC-Easy, ARC-Challenge, and OpenBookQA. The original dataset converts task-specific training examples into a unified instruction-response format using predefined templates. We used this combined dataset for commonsense fine-tuning across all compared methods and evaluated each benchmark separately, reporting their macro-average accuracy.
We will clarify the learning-rate search ranges, selection rationale, and training data composition in the revised manuscript to improve reproducibility.