acceptodds
Under review as a conference paper at ICLR 2027

Instance-Customized LoRA Generation via Discrete Priors for Efficient LLM Adaptation

Abstract

As a mainstream Parameter-Efficient Fine-Tuning (PEFT) method, Low-Rank Adaptation (LoRA) significantly reduces the optimization and deployment costs of large language models by introducing trainable low-rank matrices. Despite the progress, existing LoRA and its variants typically learn a universal adapter at the entire task or dataset level, failing to achieve fine-grained personalized adaptation for individual instances, and thus limiting the performance in complex scenarios. In this paper, we propose InLoRA, a novel dynamic, instance-customized low-rank adapter considering individual idiosyncrasies. In particular, our InLoRA first maintains a discrete codebook with vector quantization to store the prior patterns of adapters. More importantly, we retrieve from the codebook for individual instances and then utilize lightweight projections to generate instance-customized yet well-structured adapters, which would be combined with task-specific adapters for efficient fine-tuning. Extensive experiments demonstrate that our InLoRA not only outperforms state-of-the-art PEFT methods, but also matches or even surpasses the performance of full fine-tuning on multiple benchmarks. The code and pre-trained models will be made publicly available.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Parameter-Efficient Fine-Tuning, Low-Rank Adaptation, instance-customized adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cNttOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper proposes InLoRA, which combines a shared task-specific LoRA branch with an instance-conditioned branch. The latter retrieves and aggregates vectors from a learned codebook and uses lightweight projectors to generate input-dependent LoRA matrices. The codebook is initialized by clustering pretrained model features and subsequently optimized during fine-tuning. Experiments cover natural language understanding, mathematical reasoning, instruction tuning, and commonsense reasoning. The paper reports improved task performance with fewer trainable parameters than vanilla LoRA in several settings, alongside increased training and inference costs.

Critical strengths
  1. The reported results demonstrate some performance improvements across several tasks. On Mistral-7B, InLoRA reduces trainable parameters from 20.97M to 14.91M while improving GSM8K from 62.09 to 63.68 and MATH from 19.14 to 21.38. These results provide evidence that instance-conditioned adaptation can be useful under a constrained trainable-parameter budget.

  2. The component ablations provide empirical support for the codebook design. Table 16 compares the full method with continuous-feature conditioning, a frozen codebook, and random codebook initialization. On ARC-c, the full method achieves 71.59, compared with 66.38 without the codebook and 65.19 with random initialization at the same parameter count. These comparisons provide useful evidence for the contribution of the learned, semantically initialized codebook.

Critical weaknesses
  1. The practical quality–cost trade-off is not convincing. In the Llama-2-7B instruction-tuning setting, the MT-Bench score increases only from 4.91 to 4.99, while training time rises from 2 to approximately 8.4 hours and peak training memory from 8,467 to 35,065 MiB. Reported end-to-end inference throughput also decreases from 17.08 to 3.79 tokens/s. The Mistral-7B improvement over the reproduced PiSSA baseline is more substantial, but also comes with considerable overhead. Although the authors acknowledge these costs, the results do not establish when the reduction in trainable parameters justifies the additional training and deployment resources. This substantially weakens the practical motivation for the method.

  2. The experimental scope limits the evidence for relevance to current LLM adaptation. Most main experiments use old models such as RoBERTa, Llama-2, Mistral-7B, Llama-3.1-8B-Instruct and Gemma-7B on established benchmarks. It is therefore unclear whether the benefits persist with stronger contemporary backbones such as Qwen3.8 and more demanding generation tasks. The concern is not benchmark age alone, but whether the demonstrated gains extend to the settings where the additional complexity and computational cost would be worthwhile.

Questions for authors
  1. Under what practical resource constraints does InLoRA offer a preferable trade-off to static LoRA? In particular, do its quality gains persist when the baseline is given a comparable training-compute budget, rather than being compared primarily by trainable parameter count?

  2. Beyond the commonsense results in Table 13, is there evidence that the gains persist on a stronger contemporary backbone and a more demanding generation benchmark?

Reviewer_4LKCOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the "instance-agnostic" limitation of existing LoRA and task-aware LoRA variants, which learn a single adapter per task/dataset and thus cannot adapt to fine-grained per-instance heterogeneity. The authors propose InLoRA, a decoupled dual-branch architecture: a static task-specific LoRA branch (A0,B0)(A_0, B_0) anchors global task knowledge, while an instance-customized branch generates per-sample adapters (Ax,Bx)(A_x, B_x) via a learnable discrete codebook with a Top-K admixture module and two lightweight MLP projectors. A quantization loss aligns codebook and feature spaces, and a theoretical analysis (Theorem 3.1) bounds the approximation error relative to rank-rr LoRA. Experiments span GLUE (RoBERTa), mathematical reasoning (Mistral-7B, Gemma-7B), instruction tuning (Llama-2-7B, Mistral-7B), and commonsense reasoning (Llama-2-7B, Llama-3.1-8B), reporting gains over LoRA and other PEFT baselines with fewer trainable parameters.

Critical strengths
  1. Clear problem framing and motivation. The paper identifies a concrete gap—instance-agnostic adaptation within a task—and motivates it with an intuitive cognition analogy.. The three enumerated technical challenges (stability–specificity, personalization–efficiency, instance-discriminative bottleneck) map cleanly onto the proposed components.
  2. Broad empirical evaluation across task families and backbones Four distinct settings are covered: NLU, mathematical reasoning, instruction tuning, and commonsense reasoning, Multiple backbones are used, including RoBERTa-base/large, Mistral-7B, Gemma-7B, Llama-2-7B, and Llama-3.1-8B and The commonsense reasoning table includes a wide set of single-LoRA and MoE-based baselines.
Critical weaknesses
  1. Limited novelty relative to existing instance-conditioned LoRA methods. The task-shared + instance-specific decomposition is conceptually close to prior approaches that combine shared LoRA components with input-dependent adaptation. In particular, Conditional Mixture-of-LoRA (Shen et al., ACL 2024) dynamically constructs low-rank factors for each input instance, while Instance-Level Dynamic LoRAs Composition (Wang et al., EMNLP Findings 2024) selects and composes LoRA modules at the instance level. InLoRA differs by generating the instance-specific factors from a discrete codebook, but the paper does not adequately compare this mechanism with these closely related instance-conditioned alternatives under matched parameter and inference budgets. Consequently, it remains unclear how much of the reported gain comes from the proposed codebook mechanism, rather than from instance-conditioned adaptation more generally.
  2. Limited explanatory value of the theoretical analysis. Theorem 3.1 establishes a conditional improvement in optimal approximation error when (Ggen>Δr0,r)(G_{\mathrm{gen}}>\Delta_{r_0,r}), but the paper does not verify this condition in its experiments. The result therefore does not explain why InLoRA achieves the reported downstream performance gains.
Questions for authors
  1. Table 10 shows that InLoRA uses fewer trainable parameters than LoRA but requires substantially more training time and GPU memory, with lower inference throughput. Could the authors clarify which notion of efficiency the paper claims, and discuss in the main text when the parameter savings justify these computational costs?
  2. Eq. 4 and Algorithm 1 define (W1f∈Rrx×TW_1^f\in\mathbb{R}^{r_x\times T}), with (T) being the number of tokens in an input. Since different inputs can have different values of (T), they would require different numbers of columns in the same learned matrix (W1fW_1^f). How is this projector applied across variable-length inputs? If inputs are padded or truncated to a fixed length, please specify that length and explain whether padded positions are masked when generating (AxA_x).
Reviewer_wrUKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes InLoRA, an instance-customized parameter-efficient fine-tuning method that dynamically generates low-rank adapters for individual inputs. The approach combines a static task-specific LoRA branch with an instance-dependent branch based on a learnable discrete codebook, Top-K retrieval, and lightweight projection networks.

The method is evaluated across natural language understanding, mathematical reasoning, instruction tuning, and commonsense reasoning benchmarks using multiple pretrained language models. The results demonstrate improvements over standard LoRA and several PEFT baselines while reducing the number of trainable parameters. Additional ablation studies investigate the contributions of the shared adapter branch, codebook initialization, and retrieval configurations.

Critical strengths
  1. Meaningful instance-dependent adaptation framework.

The paper addresses an interesting limitation of conventional LoRA by introducing dynamically generated adapters conditioned on individual input instances. The combination of a shared task-specific branch and an instance-customized branch provides a reasonable mechanism for balancing general task knowledge with input-specific adaptation.

  1. Promising empirical results across diverse tasks.

The method demonstrates consistent improvements over standard LoRA on multiple benchmarks. For example, the average commonsense reasoning score on Llama-2-7B improves from 81.30 to 83.03 while using fewer trainable parameters. The evaluation across different models and tasks supports the potential general applicability of the approach.

Critical weaknesses
  1. Limited practical computational efficiency despite reduced trainable parameters.

Although InLoRA reduces the number of trainable parameters, it introduces substantial computational overhead. According to Table 10, training time on Llama-2-7B increases from 2.0 to approximately 8.4 hours compared with LoRA, while inference throughput decreases from 17.08 to 3.79 tokens/s. Peak GPU memory consumption also increases significantly. These results weaken the practical efficiency claims, particularly for latency-sensitive deployment scenarios.

  1. Insufficient isolation of the benefits of instance customization.

The improvements may partially arise from architectural capacity or optimization differences rather than instance customization alone. Some ablation configurations have substantially different parameter counts, and the mathematical reasoning experiments use different gradient accumulation settings across methods (Table 9). More controlled comparisons with parameter-matched static adapters and consistent optimization budgets are needed to establish the contribution of dynamic instance-conditioned adaptation.

Questions for authors
  1. Given the substantial training and inference overhead reported in Table 10, under what practical deployment conditions would InLoRA be preferable to standard LoRA?

  2. Could the authors provide a parameter-matched comparison against a static dual-branch LoRA baseline to isolate the contribution of instance customization?

  3. Could the authors clarify the different gradient accumulation configurations in Table 9 and their potential influence on the experimental comparisons?

  4. How much of the performance improvement comes specifically from discrete codebook retrieval rather than the additional nonlinear projection networks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.