- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper explores a very interesting direction for LoRA: given an LLM together with many downstream LoRA adapters, the LoRA parameters can be decomposed into a shared basis. A model can then be trained to predict these deltas, so that a new task no longer requires LoRA training. However, the experiments are not sufficiently thorough, and some of the conclusions are not solid.
Critical strengths
-
Clean and well-motivated reparameterization. The paper clearly explains why directly predicting LoRA factors is problematic: the transformation (B, A) ↦ (BG, G⁻¹A) leaves ΔW unchanged, making the factorization non-identifiable. For fixed shared bases, the coordinate representation Σ = UᵀΔWV is invariant to this ambiguity. Predicting these coordinates also substantially reduces the hypernetwork output dimension relative to predicting the full update. The formulation is simple and can potentially be incorporated into other weight-generation pipelines.
-
Empirical support for the shared-subspace assumption. The paper provides evidence beyond downstream accuracy: reconstruction error decreases as the shared-basis rank increases, and reconstructed adapters retain essentially the same average accuracy as the original adapters (73.2% versus 72.9% in Fig. 2b). The linear-feature analysis further provides a partial theoretical justification by showing recovery of a shared subspace under an exact shared-subspace assumption. Together, these results support the use of a shared-basis parameterization in the evaluated setting.
Critical weaknesses
-
Limited experimental scope All experiments use a single model family, Qwen2.5, at only two small scales: 0.5B for CSR and 1.5B for mathematics. There is no evaluation on larger models or another architecture, making it unclear whether the shared-basis structure and hypernetwork generation scale beyond this controlled setting. Moreover, DnD is the only substantial weight-generation baseline; RandOpt is used only as an ablation, while the base model and source-trained LoRAs are reference points rather than comparable generation methods. Please evaluate B2L on larger and architecturally different backbones and against additional weight-space learning baselines.
-
Test-set tuning and cross-configuration reporting The math results appear to use test-set hyperparameter tuning and different configurations across metrics. Table 7 sweeps the merge-time scale α directly on MATH and GSM8K. The headline result in Table 2, 33.7/69.3, does not correspond to one α: 33.7 MATH occurs at α=0.5, where GSM8K is 65.1 and below DnD's 66.3, whereas 69.3 GSM8K occurs at α=2.0, where MATH is 30.2. Thus, the reported row combines results selected separately for each test benchmark, while the same tuning is not reported for the baselines. Please select one α using a held-out validation set and report all methods under an equivalent selection protocol.
Questions for authors
-
Limited experimental scope All experiments use a single model family, Qwen2.5, at only two small scales: 0.5B for CSR and 1.5B for mathematics. There is no evaluation on larger models or another architecture, making it unclear whether the shared-basis structure and hypernetwork generation scale beyond this controlled setting. Moreover, DnD is the only substantial weight-generation baseline; RandOpt is used only as an ablation, while the base model and source-trained LoRAs are reference points rather than comparable generation methods. Please evaluate B2L on larger and architecturally different backbones and against additional weight-space learning baselines.
-
Test-set tuning and cross-configuration reporting The math results appear to use test-set hyperparameter tuning and different configurations across metrics. Table 7 sweeps the merge-time scale α directly on MATH and GSM8K. The headline result in Table 2, 33.7/69.3, does not correspond to one α: 33.7 MATH occurs at α=0.5, where GSM8K is 65.1 and below DnD's 66.3, whereas 69.3 GSM8K occurs at α=2.0, where MATH is 30.2. Thus, the reported row combines results selected separately for each test benchmark, while the same tuning is not reported for the baselines. Please select one α using a held-out validation set and report all methods under an equivalent selection protocol.
-
insufficiently specified "Training LoRAs" baseline The paper states that the CSR "Training LoRAs" baseline is trained on the remaining source datasets, but does not provide enough detail to reproduce or interpret it. It is unclear whether the source datasets are pooled, how samples are balanced, which checkpoint is selected, and whether its rank, scale, optimization budget, and evaluation protocol match B2L and DnD. The construction of the corresponding math baseline is also unclear. Since these results are unusually low—for example, 13.5 on BoolQ and 14.8 on MATH—the paper should fully specify this baseline and clearly distinguish source-only training from target-task LoRA fine-tuning.