- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose a novel three-step method to learn a function vector for new analogical relations. They first create characteristic function vectors for each existing relation in the dataset, finetuning these vectors on at least 10 per-relation examples. Then, for a new relation, they extract per-function-vector weights from the LM and linearly combine the existing relation vectors via a weighted sum to approximate the new relation’s function vector. They then apply two more finetuning steps to better adapt the weights to the source and the target domains’ use of the relation. The resulting analogical function vector can be injected into a language model’s foward pass to produce better analogy completions than finetuning-based baselines. The experiments first measure the impact of relation modeling with finetuned function vectors and then analogy completion with analogical function vectors.
Critical strengths
- the authors present a novel approach for improving language models’ analogy completions
- the method works without training the LM’s weights themselves and can thus be applied to all models
- their approach shows strong gains on some datasets for the selected models compared to non-finetuned function vectors
- a large set of experiments measuring the impact of relation modeling with finetuned function vectors and then analogy completion with analogical function vectors
Critical weaknesses
- gathering 10 examples first for each base relation is not easily scalable if more base relations are to be incorporated
- The evaluated models are undiverse in size and not state-of-the-art. All models were released multiple years ago:
- GPT-2 Medium (released 2019), GPT J 6B (released 2021), Llama2-7B-Chat (released 2023), Llama3.1-8B-Instruct (released 2024)
- Within their parameter range there are more modern options available. Except for the Llama3.1 model I would consider these models outdated. I encourage the authors to verify the impact of their approach for at least some newer models. E.g. Qwen3.5-9B, Phi-4, SmolLM
- Table 12 suggests that for the larger models (Llama2 and Llama3.1) the composite vector from the relation initialization step already yields similar gains as the full pipeline
- gains compared to finetuning baselines seem to diminish on larger models. Results on other and larger models would therefore be interesting
- the related work section could be expanded, especially for Activation Steering and Function Vectors section
Questions for authors
- How were the evaluated models chosen? Why weren’t newer and some larger models evaluated?
- Is the approach using non-finetuned function vectors part of your novel approach or was this already proposed in the related work?
- Why at least 10 examples per relation to obtain a finetuned function vector? How was this number identified and how does performance drop when fewer examples are available?
- Are there any base relations that are more important in the weighted sum to approximate a new relation than others? Besides the agglomerative clustering approach, how would you choose a minimal set of core relation types to most effectively model new relations?
- In 3.2 do all relations belong to only one common source domain or do they make use of all available relations in the data?
- Empirical evidence for how to set Tau?
Further comments not impacting the score:
- more descriptive caption for Figure 1. Shortly describe the steps shown.
- Figure 2 could span the entire page width to be easier to read
- in Figure 2, Phase 1 you use which easily reads as one word rather than the predicted . Consider using clearer notation here
If my questions are addressed, especially regarding the choice of models, I will increase my score.
4 comments
I would like to thank the authors for providing additional details to address my comments. While I appreciate the additional experiments with Qwen2.5-7B and Qwen3-8B, I am left wondering why, again, the authors chose older models than the latest available one, Qwen3.5-9B (specifically suggested in my review). My suggestions for more modern alternatives to their smaller models, like SmolLM or Phi-4 have also remained untested. Coupled with the weak improvements of their FFV approach for Qwen3-8B, the latest model in their experiments, I have doubts about the generalization of the FFV approach. FFVs might only lead to significant gains on smaller, older models. If that is the case, improving analogies on smaller models is still valuable but should be properly framed, instead of presenting FFVs as generally impactful for better analogy modeling. Since the model selection was and still is my main issue with the paper I will keep my score as it is.
We appreciate the follow-up from iYyi.
We investigated Qwen3.5-9B, but its hybrid architecture differs substantially from the standard Transformer architectures assumed by the original Function Vector pipeline: three of every four layers use Gated DeltaNet linear attention, while only every fourth layer uses conventional full attention. Because our causal-mediation procedure for initial FV operates over standard attention-head activations, applying it to Qwen3.5 would require adapting and validating the extraction procedure rather than directly transferring our existing implementation.
We agree that SmolLM provides a useful modern test that is compatible with the current intervention framework, and we have now run the full pipeline on two SmolLM models: SmolLM2-1.7B-Instruct and SmolLM3-3B. For each model we rebuilt the full 118-relation library (initial FVs from causal mediation, then FFV optimization) and evaluated the composite and AFV on Green, SAT and Turney with the paper's hyperparameters. FFVs are trained for 10 epochs on SmolLM2 and 5 on SmolLM3 (like the LLaMA and Qwen models). The results average five FFV seeds for SmolLM3 while SmolLM2-1.7B uses a single seed due to time constraints. Phi-4 remains untested for the same reasons.
Top-5 accuracy (%):
| Model | Method | Green near | Green far | SAT | Turney |
|---|---|---|---|---|---|
| SmolLM2-1.7B | One-shot | 60.0 | 37.5 | 30.2 | 39.0 |
| Composite | 65.0 | 52.5 | 35.8 | 39.3 | |
| AFV | 62.5 | 47.5 | 38.5 | 40.4 | |
| SmolLM3-3B | One-shot | 65.0 | 40.0 | 38.0 | 41.4 |
| Composite | 72.5 | 60.0 | 56.4 | 48.9 | |
| AFV | 70.0 | 55.0 | 55.6 | 50.0 |
For every benchmark the composite improves over the one-shot baseline, especially on the Green far-analogy and SAT datasets; the full AFV remains close to the composite. On the SAT dataset, SmolLM3 (+18.4 pp) gains more than GPT-J (+14.4 pp) and LLaMA-3.1-8B (+9.9 pp) in the paper, while Qwen3 with the strongest one-shot baseline (55.9%) has the smallest gains (+1.1 pp).
The FFVs also remain effective for the SmolLM family: on SemEval zero-shot relation completion, SmolLM3-3B improves from 1.95% without intervention and 21.67% with the initial FVs to 59.98% with FFVs.
We would also like to clarify an important distinction in the Qwen3 result. The weak incremental analogy gain on Qwen3 concerns the AFV/composition stage, not the effectiveness of the FFVs themselves. On zero-shot relation completion, Qwen3 improves from 3.68% to 87.49% on the simple-task dataset and from 0.70% to 52.70% on SemEval after FFV intervention; the corresponding initial FVs achieve only 22.98% and 5.94% respectively. Thus, learned relation vectors remain highly effective in the newer models we test. What becomes smaller for Qwen3 is the additional benefit of composing those representations for analogy, where the baseline Qwen3 model is already considerably stronger.
I appreciate the authors' quick follow-up experiments and their explanation for why Qwen3.5-9B is non-trivial to adapt. In light of the new results and the additional clarifications I will increase my score.
We thank reviewer iYyi for the insightful response.
Weaknesses
The AFV’s total supervision cost grows with the number of relations in the library, since each added relation requires examples for initialization and FFV optimization. Our claim is that high analogy performance does not require an exhaustive relation inventory. In our library-size ablation (Table 25) we reduce the 118-relation FFV library to K = {30,49,60,90} cluster representatives: performance plateaus near K=49 and only K=30 drops significantly. The SemEval sample-efficiency analysis (Table 11) shows that useful FFVs only need a few optimization pairs per relation. Thus, we view scaling as selecting a compact, diverse relational basis rather than adding every possible relation.
We extended all major evaluations to Qwen2.5-7B-Instruct and Qwen3-8B, giving six models from three families (Table 15). The composite/AFV improves over the one-shot baseline on Green far, SAT, and Turney for both models, with smaller gains for the already-strong Qwen3.
In the new stage-wise ablation (Table 26) composition is the primary source of the improvement, particularly for the larger models. Source refinement adds gains in some settings (GPT-2 and GPT-J on Turney, LLaMA-2 on SAT) with less for LLaMA-3.1 and the Qwen models. Source-target alignment adds little accuracy but reduces KL(pT∣∣w) by 6-38% on SAT and Turney while diverging from the source-derived distribution only modestly.
We have corrected the source and target assignment of the stories used for the Turney dataset. Across the GPT and LLaMA models in Table 19, the relative performance against LoFiT does not diminish with model size: LoFiT is stronger for the GPT models on Green and SAT, whereas AFV is stronger for the LLaMA models. This is not a matched comparison as LoFiT is trained on 2,898 four-term analogies in the evaluation format, whereas AFV receives no analogy training and constructs its intervention from FFVs at test time. We therefore see a trade-off between task-specific fine-tuning and inference-time relational composition. We cannot determine whether this pattern continues for models >8B.
We expanded the Related Work section to discuss task vectors (Hendel et al., 2023), in-context vectors (Liu et al., 2023), ReFT (Wu et al., 2024), LoFiT, and DyVec. These extract or train an intervention for a task specified by demonstrations while we construct one for an unlabeled relation from a reusable relation-vector library, even when its relation type is not in that library.
Questions
GPT-2-Medium tests the methods in a lightweight 355M-parameter model, GPT-J and LLaMA-2 enable direct comparison with prior Function-Vector work, and LLaMA-3.1 is a more recent 8B instruction-tuned model. All models are open-weight and expose the activations required for FV extraction. We add Qwen2.5-7B-Instruct and Qwen3-8B to expand evaluation to the Qwen family.
The initial Function Vectors are not our contribution; they follow Todd et al. (2024) and only initialize the Fine-tuned Function Vectors (FFVs). Our contributions are the relation-specific FFV optimization, the source-conditioned composition, and analogy-specific refinement. Replacing the FFVs with the original FVs for GPT-J lowers accuracy from 50.27% to 37.43% on SAT and from 29.41% to 24.60% on Turney (Table 24).
10 examples is the default budget, not a minimum. The SemEval relations have few word pairs, and ten leaves examples for held-out evaluation. With fewer pairs (Table 11) performance declines gradually, but even three examples substantially improve over the initial FV for both models.
No small fixed subset of relations dominates composition. For each analogy the weights spread over effectively 14-48 of the 118 relations, and no relation's mean weight exceeds 4.0× the uniform weight. Relation-importance rankings agree across models (Spearman r = .68-.84), suggesting shared relational structure but no dominant set. A compact library is therefore coverage-based: clustering picks representatives spanning the FFV space (Weakness 1).
All 118 relations are candidates for every analogy; they are not restricted by semantic domain or dataset of the source pair. For a source analog (xS,yS) each base relation is scored by how well its FFV predicts yS from xS, and the normalized scores are the composition weights. Thus, an analogy can draw on relations from SemEval, the original Function Vector tasks, Google, and MSR.
We fix the composition temperature τ=1 for all models and benchmarks. The sweep on SAT and Turney (Table 20) shows that lower temperatures can improve the GPT models, whereas τ=1 is near the best setting for the LLaMA and Qwen models. We retain τ=1 as a single setting without benchmark- or model-specific tuning; under a uniform prior, the weights are the normalized source likelihoods of the candidate relations.