acceptodds
Withdrawn from ICLR 2027

Basis-to-LoRA: Learning to Generate LoRA Adapters for Task Adaptation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Parameter-efficient fine-tuning methods such as Low-Rank Adapterss (LoRAs) have become a standard approach for adapting large language models to downstream tasks. Nevertheless, training a separate LoRA adapter for every new task remains computationally expensive. Weight-space learning offers an alternative by training a hypernetwork to generate adapter weights directly. However, predicting full weight updates is challenging due to their high dimensionality, while predicting LoRA factors introduces non-identifiability from invariance to a group of linear transformations. In this work, we show that LoRA updates trained on related datasets approximately share a low-dimensional subspace, and leverage this structure to improve weight-space learning. Our method learns a shared basis from a collection of trained LoRA checkpoints and constrains the hypernetwork to predict only the coordinates of a new adapter within this basis. This formulation provides an identifiable representation while substantially reducing the hypernetwork’s output dimension. We evaluate our approach on commonsense reasoning and mathematical reasoning benchmarks, where it consistently outperforms weight-space learning baselines. On Qwen2.5-1.5B, our method achieves 33.7% accuracy on MATH and 69.5% on GSM8K, improving over Drag-and-Drop by 9.8 and 3.2 percentage points, respectively.

Primary area:
foundation or frontier models, including LLMs
Keywords:
weight space learning, low-rank adaptation, LoRA, LLM adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_R7ShOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper explores a very interesting direction for LoRA: given an LLM together with many downstream LoRA adapters, the LoRA parameters can be decomposed into a shared basis. A model can then be trained to predict these deltas, so that a new task no longer requires LoRA training. However, the experiments are not sufficiently thorough, and some of the conclusions are not solid.

Critical strengths
  1. Clean and well-motivated reparameterization. The paper clearly explains why directly predicting LoRA factors is problematic: the transformation (B, A) ↦ (BG, G⁻¹A) leaves ΔW unchanged, making the factorization non-identifiable. For fixed shared bases, the coordinate representation Σ = UᵀΔWV is invariant to this ambiguity. Predicting these coordinates also substantially reduces the hypernetwork output dimension relative to predicting the full update. The formulation is simple and can potentially be incorporated into other weight-generation pipelines.

  2. Empirical support for the shared-subspace assumption. The paper provides evidence beyond downstream accuracy: reconstruction error decreases as the shared-basis rank increases, and reconstructed adapters retain essentially the same average accuracy as the original adapters (73.2% versus 72.9% in Fig. 2b). The linear-feature analysis further provides a partial theoretical justification by showing recovery of a shared subspace under an exact shared-subspace assumption. Together, these results support the use of a shared-basis parameterization in the evaluated setting.

Critical weaknesses
  1. Limited experimental scope All experiments use a single model family, Qwen2.5, at only two small scales: 0.5B for CSR and 1.5B for mathematics. There is no evaluation on larger models or another architecture, making it unclear whether the shared-basis structure and hypernetwork generation scale beyond this controlled setting. Moreover, DnD is the only substantial weight-generation baseline; RandOpt is used only as an ablation, while the base model and source-trained LoRAs are reference points rather than comparable generation methods. Please evaluate B2L on larger and architecturally different backbones and against additional weight-space learning baselines.

  2. Test-set tuning and cross-configuration reporting The math results appear to use test-set hyperparameter tuning and different configurations across metrics. Table 7 sweeps the merge-time scale α directly on MATH and GSM8K. The headline result in Table 2, 33.7/69.3, does not correspond to one α: 33.7 MATH occurs at α=0.5, where GSM8K is 65.1 and below DnD's 66.3, whereas 69.3 GSM8K occurs at α=2.0, where MATH is 30.2. Thus, the reported row combines results selected separately for each test benchmark, while the same tuning is not reported for the baselines. Please select one α using a held-out validation set and report all methods under an equivalent selection protocol.

Questions for authors
  1. Limited experimental scope All experiments use a single model family, Qwen2.5, at only two small scales: 0.5B for CSR and 1.5B for mathematics. There is no evaluation on larger models or another architecture, making it unclear whether the shared-basis structure and hypernetwork generation scale beyond this controlled setting. Moreover, DnD is the only substantial weight-generation baseline; RandOpt is used only as an ablation, while the base model and source-trained LoRAs are reference points rather than comparable generation methods. Please evaluate B2L on larger and architecturally different backbones and against additional weight-space learning baselines.

  2. Test-set tuning and cross-configuration reporting The math results appear to use test-set hyperparameter tuning and different configurations across metrics. Table 7 sweeps the merge-time scale α directly on MATH and GSM8K. The headline result in Table 2, 33.7/69.3, does not correspond to one α: 33.7 MATH occurs at α=0.5, where GSM8K is 65.1 and below DnD's 66.3, whereas 69.3 GSM8K occurs at α=2.0, where MATH is 30.2. Thus, the reported row combines results selected separately for each test benchmark, while the same tuning is not reported for the baselines. Please select one α using a held-out validation set and report all methods under an equivalent selection protocol.

  3. insufficiently specified "Training LoRAs" baseline The paper states that the CSR "Training LoRAs" baseline is trained on the remaining source datasets, but does not provide enough detail to reproduce or interpret it. It is unclear whether the source datasets are pooled, how samples are balanced, which checkpoint is selected, and whether its rank, scale, optimization budget, and evaluation protocol match B2L and DnD. The construction of the corresponding math baseline is also unclear. Since these results are unusually low—for example, 13.5 on BoolQ and 14.8 on MATH—the paper should fully specify this baseline and clearly distinguish source-only training from target-task LoRA fine-tuning.

Reviewer_WHVSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Basis-to-LoRA (B2L), a weight-space learning approach for generating task-specific LoRA adapters through low-dimensional coordinate prediction. B2L extracts shared left and right bases from a collection of pretrained LoRA checkpoints using joint diagonalization and trains a hypernetwork to predict task-specific coordinates within the resulting shared subspace. The predicted coordinates are combined with the fixed bases to reconstruct the LoRA weight updates. The experimental evaluation uses Qwen2.5-0.5B for commonsense reasoning and Qwen2.5-1.5B for mathematical reasoning, with comparisons against trained LoRA adapters and Drag-and-Drop (DnD). Additional experiments examine shared-basis reconstruction, cross-domain generalization, and comparisons with random coordinate optimization.

Critical strengths
  1. Well-motivated and structured parameterization. B2L reformulates LoRA generation as low-dimensional coordinate prediction, providing a principled approach to reducing output dimensionality while avoiding LoRA factorization ambiguity.
  2. Consistent empirical improvements. B2L outperforms DnD across all seven commonsense reasoning datasets and both mathematical reasoning benchmarks. The reconstruction experiments further demonstrate that shared bases can preserve adapter performance.
Critical weaknesses
  1. Insufficient validation of the core methodological contribution. The shared-basis construction builds on existing joint-diagonalization methods, while the hypernetwork follows DnD. The paper lacks comparisons with alternative low-dimensional representations, making it difficult to isolate the contribution of the proposed parameterization.
  2. Limited efficiency and generalization evidence. Despite emphasizing computational efficiency, the paper does not report basis construction costs, training time, or inference latency. Experiments are restricted to Qwen2.5 models and standardized LoRA configurations, limiting the evidence for broader applicability.
Questions for authors
  1. How does B2L compare with PCA- or SVD-based coordinate prediction under identical training conditions?
  2. How sensitive is downstream performance to the shared-basis rank and reconstruction threshold?
  3. What are the computational costs of basis construction, hypernetwork training, and adapter generation compared with DnD?
  4. Can the authors clarify the cross-domain evaluation protocol and the inconsistent GSM8K results?
Reviewer_k56KOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Basis-to-LoRA (B2L), a weight-space learning method for generating task-specific LoRA adapters without new training. It learns the shared bases from a collection of already trained LoRA adapters using joint diagonalization. Then a hypernetwork is trained to predict task-specific coordinates on these bases. Experiments on commonsense and math reasoning show that the proposed B2L outperforms the baselines.

Critical strengths

The strengths are summarized below:

  1. The proposed method is straightforward and effective. The presentation is clear.
Critical weaknesses

The weaknesses are summarized below:

  1. The novelty is limited. The shared-basis in Eq. (9), the hypernet, the weight prediction framework are largely adopt from existing works. The primary new idea is to predict low-dimensional coordinates instead the LoRA parameters. The paper does not clearly establish its own novelty beyond the existing techniques.
  2. The theory section looks trivial, and is not sufficiently connected to the proposed method.
  3. The manuscript directly cites many results from the original DnD paper [1]. However, the reported DnD accuracy of 47% on PIQA in Table 1, is inconsistent with the original result, 87.9% accuracy. It would be better that, the authors provides a clear clarification about the results.
  4. Regarding the baseline, trained LoRA. I think it is unfair. For B2L, it constructs update ΔW^=UΣ^V⊤\widehat{\Delta W}=U\hat{\Sigma}V^\top, which can have high ranks, even the source adapters U,VU,V are low rank. It is not clear whether the improvements against LoRA from the proposed framework, a change in rank, or something else.

[1] Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights. https://arxiv.org/abs/2506.16406.

Questions for authors

See above weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.