Adaptive Task Vectors for Large Language Models

In-Context Learning (ICL) enables Large Language Models (LLMs) to perform tasks without parameter updates by conditioning on demonstrations in the prompt. Despite its success, ICL suffers from sensitivity to demonstration order, context length constraints, and computational inefficiency. Task vector-based approaches address these issues by compressing task information into a single vector. However, existing methods typically construct task vectors from fixed demonstration sets and reuse them across queries, limiting their ability to adapt to individual inputs and generalize to unseen tasks. We propose Adaptive Task Vectors (ATV), a framework that dynamically generates task vectors conditioned on each input query. ATV employs a small language model whose last-token and mean-pooled representations are jointly projected into task vectors and injected into the frozen target LLM's hidden states. This enables adaptive model steering without explicit demonstrations during inference. Theoretically, ATV extends the static LoRA-style residual perspective via query conditioning and introduces interactions beyond Prefix-Tuning via direct hidden-state injection. Empirically, ATV achieves strong in-domain and unseen-task performance, including an adversarial shift, demonstrating effective adaptive model steering.