Exemplar-Derived Soft Prompts for Efficient and Effective Domain Adaptation

Adapting foundation models to new domains is challenging and computationally expensive. While parameter-efficient fine-tuning (PEFT) methods allow models to acquire domain-specific skills, they require updating deployed models and incur additional training and deployment overhead. In contrast, In-context Learning (ICL) avoids model updates and improves over off-the-shelf models by leveraging similar exemplars, but it often fails to achieve competitive accuracy on specialised tasks. Motivated by the success of in-context exemplars and the need for fine-tuning-level adaptation, we propose Multi-Head Attention-based Exemplar Soft Prompting (MHA-ESP), which uses an attention mechanism to learn soft prompts from retrieved exemplars, with multiple attention heads controlling prompt generation. Across multiple benchmarks and model scales, MHA-ESP performs on par with Low-Rank Adaptation (LoRA), outperforms standard ICL by an average of 18.85 points, and reduces inference cost by up to 10×GFLOPs, enabling efficient, high- accuracy domain adaptation without updating the foundation model