Extracting and Composing Function Vectors for Analogical Reasoning over Latent Relations

Analogical reasoning requires models to abstract relational structure and transfer it across domains, yet how large language models internally represent and manipulate such relations remains poorly understood. Recent work on mechanistic interpretability suggests that in-context tasks can be compressed into compact activation-space representations, or function vectors, but their capacity to support compositional and out-of-distribution reasoning has received limited study. We introduce a framework for learning, composing, and refining relation-specific function vectors for analogical reasoning. First, we optimize function vectors using as few as 10 relational examples while keeping the language model frozen. These fine-tuned function vectors substantially improve zero-shot relation completion and exhibit stronger alignment with human judgments of relational similarity. Next, we show that previously unseen relational concepts can be represented as weighted linear combinations of learned function vectors, enabling transfer to novel analogies. Finally, we introduce Analogical Function Vectors (AFVs), which refine these composition weights at inference time using a relation-consistency objective between source and target domains. Injecting AFVs into model activations improves performance on four-term analogies involving out-of-distribution relations not used to derive the function vectors, SAT analogy problems, and narrative analogy tasks across models ranging from GPT-2 to LLaMA-3.1-8B. Our results suggest that relational knowledge in language models can be represented and manipulated through compact activation-space structures, providing a parameter-efficient and mechanistically interpretable approach to improving analogical reasoning.