STAGE: Scaling Task-Specific LoRA Generation from Few-Shot Examples

Parameter-efficient fine-tuning reduces the cost of adapting pretrained models, but still requires a separate optimization process for every new task. We study whether previously optimized task adapters can serve as supervision for generating a new adapter from few-shot examples, and how generation changes as this supervision grows. We introduce STAGE, a conditional Flow Matching framework that generates LoRA adapters directly from support examples without gradient-based adapter fitting at deployment. For each training task, we pair a small labeled support set with an independently optimized adapter. We represent the adapter through SVD-based factors of its effective weight update to reduce ambiguity in the original LoRA factors. For image classification, we construct up to 100,000 fitted adapters per dataset-backbone pair, and compare bank sizes of 1K, 10K, and 100K. At a fixed generation budget, support-selected accuracy increases with bank size in all six dataset-backbone settings. With only a few support images, generated adapters improve official-split accuracy over the frozen classifier by up to approximately 13.4 and 9.9 percentage points on CIFAR-100 and Tiny-ImageNet, respectively. We also evaluate a text generator on 14 language-generation datasets, with accuracy gains of 31.7 and 21.9 percentage points on e-SNLI and WikiSQL, respectively. These results show that scaling task-adapter supervision can improve few-shot adaptation without gradient-based adapter fitting at deployment.