acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Task Vectors for Large Language Models

Abstract

In-Context Learning (ICL) enables Large Language Models (LLMs) to perform tasks without parameter updates by conditioning on demonstrations in the prompt. Despite its success, ICL suffers from sensitivity to demonstration order, context length constraints, and computational inefficiency. Task vector-based approaches address these issues by compressing task information into a single vector. However, existing methods typically construct task vectors from fixed demonstration sets and reuse them across queries, limiting their ability to adapt to individual inputs and generalize to unseen tasks. We propose Adaptive Task Vectors (ATV), a framework that dynamically generates task vectors conditioned on each input query. ATV employs a small language model whose last-token and mean-pooled representations are jointly projected into task vectors and injected into the frozen target LLM's hidden states. This enables adaptive model steering without explicit demonstrations during inference. Theoretically, ATV extends the static LoRA-style residual perspective via query conditioning and introduces interactions beyond Prefix-Tuning via direct hidden-state injection. Empirically, ATV achieves strong in-domain and unseen-task performance, including an adversarial shift, demonstrating effective adaptive model steering.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
In-context Learning, Task Vectors, Large Language Models, Query-dependent

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FXBSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Adaptive Task Vectors (ATV), which uses a small language model to generate query-specific task vectors and injects them into a frozen LLM. Unlike previous methods that rely on fixed or retrieved vectors, ATV dynamically generates vectors for each input. Experiments show promising results on both in-domain and unseen tasks. The paper also provides theoretical connections to LoRA and Prefix-Tuning.

Critical strengths
  • ATV consistently outperforms several baselines across different models and tasks. The unseen-task and adversarial evaluations are particularly interesting.

  • Generating task vectors directly from queries is intuitive and avoids relying on a fixed vector library. The paper is also generally clear and includes useful ablations.

Critical weaknesses
  • Novelty is somewhat limited. Query-conditioned steering has been explored in prior work, such as HyperSteer and ICR. The main contribution seems to be combining existing ideas, but the paper does not directly compare against these closely related methods.

  • The comparisons are not fully controlled. ATV uses a trained generator with around 36M trainable parameters, while several baselines have very different training setups and capacities. It is therefore unclear how much of the improvement comes specifically from query conditioning. Also, the theoretical results mainly establish connections to LoRA and Prefix-Tuning under simplified assumptions, rather than proving a clear advantage of ATV.

Questions for authors
  • Could the authors compare ATV with more closely related query-conditioned steering methods, such as HyperSteer or ICR?

  • Could the authors provide a parameter-matched comparison between static and query-conditioned task vectors? This would help isolate the actual benefit of dynamic generation.

  • The LoRA equivalence result considers a generalized static ATV formulation. How does this theoretical result help explain the advantage of the actual query-conditioned ATV?

Reviewer_yJnJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Adaptive Task Vectors (ATV), a framework for steering frozen large language models using query-conditioned task vectors. Unlike prior task-vector approaches that reuse fixed vectors extracted from demonstration prompts or retrieve vectors from a precomputed library, ATV dynamically generates a distinct set of layer-wise vectors for each input query. Specifically, a small language model encodes the query, and a Last-mean pooling combines its last-token and mean-pooled representations to produce vectors for all layers of the target LLM. These vectors are additively injected into the last-token hidden state of the frozen target model. During training, only the generator’s LoRA parameters and the expansion module are optimized using the target model’s cross-entropy loss together with an i-Mix regularization objective based on dataset identity; no demonstrations or dataset labels are required at inference time.

Critical strengths

1.The paper addresses a clear limitation of existing task-vector methods by generating query-conditioned, layer-wise steering vectors rather than reusing fixed demonstration-derived vectors. 2.ATV keeps the target LLM frozen, does not increase the target-model input length, and can be integrated through additive hidden-state intervention. 3.The representation and layer-wise analyses are consistent with the proposed motivation and provide useful insights into the behavior of the generated vectors.

Critical weaknesses

1.Although the vector additions themselves are computationally lightweight, it requires access to intermediate hidden states and query-specific interventions across Transformer layers. Not directly plug-and-play for existing LLM inference systems. 2.The paper emphasizes demonstration-free inference but does not sufficiently account for the offline training cost or clarify that ATV still depends on substantial demonstration supervision during training.

Questions for authors

For the unseen-task evaluation, what is the candidate demonstration pool used by BM25? Does it include demonstrations from the held-out task itself, or only examples from the in-domain training tasks? The paper uses Prefix-Tuning as a key theoretical reference but does not include it as an empirical baseline. A parameter-matched Prefix-Tuning comparison under the same training data and target backbones would help determine whether ATV’s additional cross-interaction terms translate into practical performance gains.

Reviewer_wEN1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Adaptive Task Vectors (ATV), a framework for query-conditioned activation steering of frozen large language models. Unlike conventional task-vector methods that reuse representations extracted from fixed demonstration sets, ATV generates layer-wise steering vectors dynamically for each input query. Specifically, a small language model produces last-token and mean-pooled representations, which are combined through a trainable projection module and injected into the target LLM's hidden states. The generator is optimized using target-model cross-entropy and an auxiliary i-Mix regularization objective.

The paper also presents theoretical connections to LoRA-style residual adaptation and Prefix-Tuning. Empirically, the authors evaluate ATV on Llama-3-8B and Mistral-7B across 20 in-domain tasks, five unseen tasks, additional out-of-distribution benchmarks, and the adversarial HANS dataset. The experiments demonstrate promising accuracy improvements over existing task-vector and in-context learning baselines, supported by ablations on generator architecture, representation construction, and injection strategies.

Critical strengths
  1. Strong empirical results across multiple evaluation settings.

ATV achieves substantial improvements over existing task-vector baselines on both Llama-3-8B and Mistral-7B. In particular, it improves the average in-domain accuracy from 58.3% (BM25) to 66.6% on Llama-3, and from 55.5% to 61.2% on Mistral. The paper additionally evaluates unseen tasks, benchmarks excluded from training, and the adversarial HANS dataset. These results provide encouraging evidence that the approach can transfer beyond its training tasks. The breadth of the evaluation and the consistency of the main improvements are important strengths.

  1. A simple and practical approach to query-conditioned model steering.

The proposed framework generates layer-wise steering vectors directly from input queries without requiring demonstrations or retrieval during inference. The target LLM remains frozen, making the method potentially useful for adapting existing models. The paper also provides meaningful ablations of generator capacity, architecture, injection layers, Last-Mean pooling, and i-Mix regularization. In particular, the comparisons with static task-vector variants in Appendix K provide initial evidence supporting the use of query-conditioned generation.

Critical weaknesses
  1. Insufficient comparison with closely related methods.

ATV introduces additional supervised training and approximately 36M trainable parameters, while most main baselines use different training mechanisms and parameter budgets. More controlled comparisons with supervised LoRA and existing query-conditioned steering methods would better establish the benefits of ATV.

  1. Limited theoretical justification.

The theoretical analysis mainly considers generalized or simplified settings rather than the actual ATV implementation. In particular, the LoRA equivalence applies to a generalized affine formulation, while the Prefix-Tuning comparison relies on simplified linear attention. Their implications for ATV's practical advantages remain unclear.

Questions for authors

Q1. Comparison with supervision-matched baselines.

Could the authors provide more detailed in-domain and unseen-task results for the supervised LoRA baseline shown in Figure 5, ideally under matched training data and optimization budgets? How does ATV compare with more closely related query-conditioned approaches, such as ICR or other input-conditioned steering methods? Such evidence would materially affect my assessment of the method's novelty and empirical advantage.

Q2. Isolating the contribution of input conditioning.

Appendix K compares ATV with directly optimized static vectors, but these alternatives have substantially lower trainable capacity. Could the authors provide a stronger capacity-controlled ablation, such as an input-agnostic generator with comparable trainable parameters or a control that disrupts the query-vector correspondence? This would help distinguish the benefit of query-specific adaptation from the additional capacity and supervised training.

Q3. Scope of the theoretical and generalization claims.

Could the authors clarify what the theoretical results imply specifically for the implemented additive ATV, rather than its generalized affine formulation? In addition, the unseen-task improvement over BM25 on Llama-3 has a reported one-sided p-value of 0.0568 based on three seeds (Appendix G). Could the authors provide additional uncertainty estimates or a more detailed statistical analysis to substantiate the robustness of these generalization gains?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.