- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper proposes InLoRA, which combines a shared task-specific LoRA branch with an instance-conditioned branch. The latter retrieves and aggregates vectors from a learned codebook and uses lightweight projectors to generate input-dependent LoRA matrices. The codebook is initialized by clustering pretrained model features and subsequently optimized during fine-tuning. Experiments cover natural language understanding, mathematical reasoning, instruction tuning, and commonsense reasoning. The paper reports improved task performance with fewer trainable parameters than vanilla LoRA in several settings, alongside increased training and inference costs.
Critical strengths
-
The reported results demonstrate some performance improvements across several tasks. On Mistral-7B, InLoRA reduces trainable parameters from 20.97M to 14.91M while improving GSM8K from 62.09 to 63.68 and MATH from 19.14 to 21.38. These results provide evidence that instance-conditioned adaptation can be useful under a constrained trainable-parameter budget.
-
The component ablations provide empirical support for the codebook design. Table 16 compares the full method with continuous-feature conditioning, a frozen codebook, and random codebook initialization. On ARC-c, the full method achieves 71.59, compared with 66.38 without the codebook and 65.19 with random initialization at the same parameter count. These comparisons provide useful evidence for the contribution of the learned, semantically initialized codebook.
Critical weaknesses
-
The practical quality–cost trade-off is not convincing. In the Llama-2-7B instruction-tuning setting, the MT-Bench score increases only from 4.91 to 4.99, while training time rises from 2 to approximately 8.4 hours and peak training memory from 8,467 to 35,065 MiB. Reported end-to-end inference throughput also decreases from 17.08 to 3.79 tokens/s. The Mistral-7B improvement over the reproduced PiSSA baseline is more substantial, but also comes with considerable overhead. Although the authors acknowledge these costs, the results do not establish when the reduction in trainable parameters justifies the additional training and deployment resources. This substantially weakens the practical motivation for the method.
-
The experimental scope limits the evidence for relevance to current LLM adaptation. Most main experiments use old models such as RoBERTa, Llama-2, Mistral-7B, Llama-3.1-8B-Instruct and Gemma-7B on established benchmarks. It is therefore unclear whether the benefits persist with stronger contemporary backbones such as Qwen3.8 and more demanding generation tasks. The concern is not benchmark age alone, but whether the demonstrated gains extend to the settings where the additional complexity and computational cost would be worthwhile.
Questions for authors
-
Under what practical resource constraints does InLoRA offer a preferable trade-off to static LoRA? In particular, do its quality gains persist when the baseline is given a comparable training-compute budget, rather than being compared primarily by trainable parameter count?
-
Beyond the commonsense results in Table 13, is there evidence that the gains persist on a stronger contemporary backbone and a more demanding generation benchmark?