- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Adaptive Task Vectors (ATV), which uses a small language model to generate query-specific task vectors and injects them into a frozen LLM. Unlike previous methods that rely on fixed or retrieved vectors, ATV dynamically generates vectors for each input. Experiments show promising results on both in-domain and unseen tasks. The paper also provides theoretical connections to LoRA and Prefix-Tuning.
Critical strengths
-
ATV consistently outperforms several baselines across different models and tasks. The unseen-task and adversarial evaluations are particularly interesting.
-
Generating task vectors directly from queries is intuitive and avoids relying on a fixed vector library. The paper is also generally clear and includes useful ablations.
Critical weaknesses
-
Novelty is somewhat limited. Query-conditioned steering has been explored in prior work, such as HyperSteer and ICR. The main contribution seems to be combining existing ideas, but the paper does not directly compare against these closely related methods.
-
The comparisons are not fully controlled. ATV uses a trained generator with around 36M trainable parameters, while several baselines have very different training setups and capacities. It is therefore unclear how much of the improvement comes specifically from query conditioning. Also, the theoretical results mainly establish connections to LoRA and Prefix-Tuning under simplified assumptions, rather than proving a clear advantage of ATV.
Questions for authors
-
Could the authors compare ATV with more closely related query-conditioned steering methods, such as HyperSteer or ICR?
-
Could the authors provide a parameter-matched comparison between static and query-conditioned task vectors? This would help isolate the actual benefit of dynamic generation.
-
The LoRA equivalence result considers a generalized static ATV formulation. How does this theoretical result help explain the advantage of the actual query-conditioned ATV?