acceptodds
Under review as a conference paper at ICLR 2027

Extracting and Composing Function Vectors for Analogical Reasoning over Latent Relations

Abstract

Analogical reasoning requires models to abstract relational structure and transfer it across domains, yet how large language models internally represent and manipulate such relations remains poorly understood. Recent work on mechanistic interpretability suggests that in-context tasks can be compressed into compact activation-space representations, or function vectors, but their capacity to support compositional and out-of-distribution reasoning has received limited study. We introduce a framework for learning, composing, and refining relation-specific function vectors for analogical reasoning. First, we optimize function vectors using as few as 10 relational examples while keeping the language model frozen. These fine-tuned function vectors substantially improve zero-shot relation completion and exhibit stronger alignment with human judgments of relational similarity. Next, we show that previously unseen relational concepts can be represented as weighted linear combinations of learned function vectors, enabling transfer to novel analogies. Finally, we introduce Analogical Function Vectors (AFVs), which refine these composition weights at inference time using a relation-consistency objective between source and target domains. Injecting AFVs into model activations improves performance on four-term analogies involving out-of-distribution relations not used to derive the function vectors, SAT analogy problems, and narrative analogy tasks across models ranging from GPT-2 to LLaMA-3.1-8B. Our results suggest that relational knowledge in language models can be represented and manipulated through compact activation-space structures, providing a parameter-efficient and mechanistically interpretable approach to improving analogical reasoning.

Primary area:
interpretability and explainable AI
Keywords:
Large Language Models, Mechanistic Interpretability, Activation Patching

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_iYyiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a novel three-step method to learn a function vector for new analogical relations. They first create characteristic function vectors for each existing relation in the dataset, finetuning these vectors on at least 10 per-relation examples. Then, for a new relation, they extract per-function-vector weights from the LM and linearly combine the existing relation vectors via a weighted sum to approximate the new relation’s function vector. They then apply two more finetuning steps to better adapt the weights to the source and the target domains’ use of the relation. The resulting analogical function vector can be injected into a language model’s foward pass to produce better analogy completions than finetuning-based baselines. The experiments first measure the impact of relation modeling with finetuned function vectors and then analogy completion with analogical function vectors.

Critical strengths
  • the authors present a novel approach for improving language models’ analogy completions
  • the method works without training the LM’s weights themselves and can thus be applied to all models
  • their approach shows strong gains on some datasets for the selected models compared to non-finetuned function vectors
  • a large set of experiments measuring the impact of relation modeling with finetuned function vectors and then analogy completion with analogical function vectors
Critical weaknesses
  • gathering 10 examples first for each base relation is not easily scalable if more base relations are to be incorporated
  • The evaluated models are undiverse in size and not state-of-the-art. All models were released multiple years ago:
    • GPT-2 Medium (released 2019), GPT J 6B (released 2021), Llama2-7B-Chat (released 2023), Llama3.1-8B-Instruct (released 2024)
    • Within their parameter range there are more modern options available. Except for the Llama3.1 model I would consider these models outdated. I encourage the authors to verify the impact of their approach for at least some newer models. E.g. Qwen3.5-9B, Phi-4, SmolLM
  • Table 12 suggests that for the larger models (Llama2 and Llama3.1) the composite vector from the relation initialization step already yields similar gains as the full pipeline
  • gains compared to finetuning baselines seem to diminish on larger models. Results on other and larger models would therefore be interesting
  • the related work section could be expanded, especially for Activation Steering and Function Vectors section
Questions for authors
  • How were the evaluated models chosen? Why weren’t newer and some larger models evaluated?
  • Is the approach using non-finetuned function vectors part of your novel approach or was this already proposed in the related work?
  • Why at least 10 examples per relation to obtain a finetuned function vector? How was this number identified and how does performance drop when fewer examples are available?
  • Are there any base relations that are more important in the weighted sum to approximate a new relation than others? Besides the agglomerative clustering approach, how would you choose a minimal set of core relation types to most effectively model new relations?
  • In 3.2 do all relations z∈Zz \in Z belong to only one common source domain or do they make use of all available relations in the data?
  • Empirical evidence for how to set Tau?

Further comments not impacting the score:

  • more descriptive caption for Figure 1. Shortly describe the steps shown.
  • Figure 2 could span the entire page width to be easier to read
  • in Figure 2, Phase 1 you use ySpredy_{Spred} which easily reads as one word rather than the predicted ySy_S. Consider using clearer notation here

If my questions are addressed, especially regarding the choice of models, I will increase my score.

4 comments

We thank reviewer iYyi for the insightful response.

Weaknesses

  1. The AFV’s total supervision cost grows with the number of relations in the library, since each added relation requires examples for initialization and FFV optimization. Our claim is that high analogy performance does not require an exhaustive relation inventory. In our library-size ablation (Table 25) we reduce the 118-relation FFV library to K = {30,49,60,90} cluster representatives: performance plateaus near K=49 and only K=30 drops significantly. The SemEval sample-efficiency analysis (Table 11) shows that useful FFVs only need a few optimization pairs per relation. Thus, we view scaling as selecting a compact, diverse relational basis rather than adding every possible relation.

  2. We extended all major evaluations to Qwen2.5-7B-Instruct and Qwen3-8B, giving six models from three families (Table 15). The composite/AFV improves over the one-shot baseline on Green far, SAT, and Turney for both models, with smaller gains for the already-strong Qwen3.

  3. In the new stage-wise ablation (Table 26) composition is the primary source of the improvement, particularly for the larger models. Source refinement adds gains in some settings (GPT-2 and GPT-J on Turney, LLaMA-2 on SAT) with less for LLaMA-3.1 and the Qwen models. Source-target alignment adds little accuracy but reduces KL(pT∣∣w)KL(p_T||w) by 6-38% on SAT and Turney while diverging from the source-derived distribution only modestly.

  4. We have corrected the source and target assignment of the stories used for the Turney dataset. Across the GPT and LLaMA models in Table 19, the relative performance against LoFiT does not diminish with model size: LoFiT is stronger for the GPT models on Green and SAT, whereas AFV is stronger for the LLaMA models. This is not a matched comparison as LoFiT is trained on 2,898 four-term analogies in the evaluation format, whereas AFV receives no analogy training and constructs its intervention from FFVs at test time. We therefore see a trade-off between task-specific fine-tuning and inference-time relational composition. We cannot determine whether this pattern continues for models >8B.

  5. We expanded the Related Work section to discuss task vectors (Hendel et al., 2023), in-context vectors (Liu et al., 2023), ReFT (Wu et al., 2024), LoFiT, and DyVec. These extract or train an intervention for a task specified by demonstrations while we construct one for an unlabeled relation from a reusable relation-vector library, even when its relation type is not in that library.

Questions

  1. GPT-2-Medium tests the methods in a lightweight 355M-parameter model, GPT-J and LLaMA-2 enable direct comparison with prior Function-Vector work, and LLaMA-3.1 is a more recent 8B instruction-tuned model. All models are open-weight and expose the activations required for FV extraction. We add Qwen2.5-7B-Instruct and Qwen3-8B to expand evaluation to the Qwen family.

  2. The initial Function Vectors are not our contribution; they follow Todd et al. (2024) and only initialize the Fine-tuned Function Vectors (FFVs). Our contributions are the relation-specific FFV optimization, the source-conditioned composition, and analogy-specific refinement. Replacing the FFVs with the original FVs for GPT-J lowers accuracy from 50.27% to 37.43% on SAT and from 29.41% to 24.60% on Turney (Table 24).

  3. 10 examples is the default budget, not a minimum. The SemEval relations have few word pairs, and ten leaves examples for held-out evaluation. With fewer pairs (Table 11) performance declines gradually, but even three examples substantially improve over the initial FV for both models.

  4. No small fixed subset of relations dominates composition. For each analogy the weights spread over effectively 14-48 of the 118 relations, and no relation's mean weight exceeds 4.0× the uniform weight. Relation-importance rankings agree across models (Spearman r = .68-.84), suggesting shared relational structure but no dominant set. A compact library is therefore coverage-based: clustering picks representatives spanning the FFV space (Weakness 1).

  5. All 118 relations are candidates for every analogy; they are not restricted by semantic domain or dataset of the source pair. For a source analog (xS,ySx_S,y_S) each base relation is scored by how well its FFV predicts ySy_S from xSx_S, and the normalized scores are the composition weights. Thus, an analogy can draw on relations from SemEval, the original Function Vector tasks, Google, and MSR.

  6. We fix the composition temperature τ=1\tau=1 for all models and benchmarks. The sweep on SAT and Turney (Table 20) shows that lower temperatures can improve the GPT models, whereas τ=1\tau=1 is near the best setting for the LLaMA and Qwen models. We retain τ=1\tau=1 as a single setting without benchmark- or model-specific tuning; under a uniform prior, the weights are the normalized source likelihoods of the candidate relations.

I would like to thank the authors for providing additional details to address my comments. While I appreciate the additional experiments with Qwen2.5-7B and Qwen3-8B, I am left wondering why, again, the authors chose older models than the latest available one, Qwen3.5-9B (specifically suggested in my review). My suggestions for more modern alternatives to their smaller models, like SmolLM or Phi-4 have also remained untested. Coupled with the weak improvements of their FFV approach for Qwen3-8B, the latest model in their experiments, I have doubts about the generalization of the FFV approach. FFVs might only lead to significant gains on smaller, older models. If that is the case, improving analogies on smaller models is still valuable but should be properly framed, instead of presenting FFVs as generally impactful for better analogy modeling. Since the model selection was and still is my main issue with the paper I will keep my score as it is.

We appreciate the follow-up from iYyi.

We investigated Qwen3.5-9B, but its hybrid architecture differs substantially from the standard Transformer architectures assumed by the original Function Vector pipeline: three of every four layers use Gated DeltaNet linear attention, while only every fourth layer uses conventional full attention. Because our causal-mediation procedure for initial FV operates over standard attention-head activations, applying it to Qwen3.5 would require adapting and validating the extraction procedure rather than directly transferring our existing implementation.

We agree that SmolLM provides a useful modern test that is compatible with the current intervention framework, and we have now run the full pipeline on two SmolLM models: SmolLM2-1.7B-Instruct and SmolLM3-3B. For each model we rebuilt the full 118-relation library (initial FVs from causal mediation, then FFV optimization) and evaluated the composite and AFV on Green, SAT and Turney with the paper's hyperparameters. FFVs are trained for 10 epochs on SmolLM2 and 5 on SmolLM3 (like the LLaMA and Qwen models). The results average five FFV seeds for SmolLM3 while SmolLM2-1.7B uses a single seed due to time constraints. Phi-4 remains untested for the same reasons.

Top-5 accuracy (%):

ModelMethodGreen nearGreen farSATTurney
SmolLM2-1.7BOne-shot60.037.530.239.0
Composite65.052.535.839.3
AFV62.547.538.540.4
SmolLM3-3BOne-shot65.040.038.041.4
Composite72.560.056.448.9
AFV70.055.055.650.0

For every benchmark the composite improves over the one-shot baseline, especially on the Green far-analogy and SAT datasets; the full AFV remains close to the composite. On the SAT dataset, SmolLM3 (+18.4 pp) gains more than GPT-J (+14.4 pp) and LLaMA-3.1-8B (+9.9 pp) in the paper, while Qwen3 with the strongest one-shot baseline (55.9%) has the smallest gains (+1.1 pp).

The FFVs also remain effective for the SmolLM family: on SemEval zero-shot relation completion, SmolLM3-3B improves from 1.95% without intervention and 21.67% with the initial FVs to 59.98% with FFVs.

We would also like to clarify an important distinction in the Qwen3 result. The weak incremental analogy gain on Qwen3 concerns the AFV/composition stage, not the effectiveness of the FFVs themselves. On zero-shot relation completion, Qwen3 improves from 3.68% to 87.49% on the simple-task dataset and from 0.70% to 52.70% on SemEval after FFV intervention; the corresponding initial FVs achieve only 22.98% and 5.94% respectively. Thus, learned relation vectors remain highly effective in the newer models we test. What becomes smaller for Qwen3 is the additional benefit of composing those representations for analogy, where the baseline Qwen3 model is already considerably stronger.

I appreciate the authors' quick follow-up experiments and their explanation for why Qwen3.5-9B is non-trivial to adapt. In light of the new results and the additional clarifications I will increase my score.

Reviewer_1iU6OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework for extracting, fine-tuning, composing, and test-time refining function vectors for analogical reasoning in LLMs. The authors first train relation-specific function vectors with limited supervision, then combine them linearly to represent unseen relations, and finally refine the mixture weights at inference time to form Analogical Function Vectors. Experiments across several models and analogy benchmarks show consistent improvements over standard prompting and some activation/fine-tuning baselines.

Critical strengths
  • Interesting and relevant problem. The paper addresses how LLMs internally represent relational knowledge and how such representations can be reused for analogical reasoning, which is important for both reasoning and interpretability.
  • Clear methodological pipeline. The progression from initial function vectors to fine-tuned function vectors, composite vectors, and AFVs is conceptually coherent.
  • Strong empirical coverage. The method is evaluated on multiple models, including GPT-2, GPT-J, LLaMA-2, and LLaMA-3.1, and on several analogy datasets, including SemEval, Green’s analogies, SAT analogies, and Turney-style narrative analogies.
  • Useful OOD evaluation. The Leave-One-Type-Out and Leave-Two-Types-Out experiments are a good attempt to test whether unseen relations can be approximated compositionally from a learned relation-vector library.
  • Good ablations and comparisons. The paper includes comparisons to ActAdd, LoRA, LoFiT, DyVec, initial FVs, randomly initialized vectors, and different library sizes.
  • Human-alignment analysis is valuable. The correlation between model relational dissimilarity matrices and human judgments provides additional evidence that the learned vectors capture meaningful relational structure.
Critical weaknesses
  • AFV gains over composite vectors are sometimes small. In several settings, especially for larger models, the composite vector already gives most of the improvement, and the additional test-time refinement provides limited benefit.
  • Evaluation relies heavily on top-5 accuracy. While justified by lexical variability, top-5 accuracy may overstate practical analogy-solving ability. More top-1 results or human-validated alternative answers would strengthen the conclusions.
  • Narrative analogy evaluation is limited. The narrative setting still uses pre-provided keywords and structured QA prompts, so it does not fully test analogical reasoning in realistic long-form narratives.
  • Potential data contamination is not deeply addressed. The analogy benchmarks are relatively old and may appear in pretraining corpora, especially for larger models.
  • Test-time cost is nontrivial. AFV requires many forward passes and sometimes forward-backward optimization per analogy, which may limit scalability.
  • The role of source-target alignment is not fully isolated. The results suggest composition and source refinement are the main drivers, while the final alignment phase has unclear incremental value.
Questions for authors
  1. How much of the improvement comes from the fine-tuned relation-vector library versus the AFV test-time refinement? Could the authors provide a clearer ablation separating initialization, source refinement, and source-target alignment?
  2. Can the method be evaluated with top-1 accuracy or with a semantic-answer matching metric, especially for SAT and Green’s analogy datasets?
  3. How sensitive are the results to the choice of injection layer, number of heads, pruning threshold, and temperature parameters?
  4. Are the old analogy benchmarks likely to be memorized by the pretrained models? Could the authors test on newly generated or adversarial analogies?
  5. For narrative analogies, how would the method perform if the relevant keywords were not explicitly provided?
  6. Does the learned FFV library transfer across models, or must each model have its own separately learned function-vector basis?
  7. Can the authors provide qualitative examples where AFV helps and where it hurts, especially for class-inclusion relations where performance decreases?
2 comments

We thank reviewer 1iU6 for the helpful response.

Weaknesses

  1. We agree that composition accounts for the majority of the AFV's accuracy gains, particularly for the larger models, and we have revised the paper to make this the primary result. However, source refinement provides additional benefits in several settings: on Turney, it improves the composite by +5.6 pp for GPT-2 (p=.002) and +7.8 pp for GPT-J (p=.002), and on SAT by +2.7 pp for LLaMA-2 (p=.031). For the larger models these gains are smaller or absent, so we now characterize refinement as a model-dependent extension as shown in the stage-wise ablation (Table 26).

  2. We report Top-1 accuracy in Table 16. The pattern is not specific to Top-5: the AFV also exceeds the one-shot baseline in Top-1 accuracy across all six models on the three main evaluations, though refinement does not uniformly improve over the composite. We retain Top-5 as the primary metric because the generative task can admit lexical competition.

  3. We now describe this experiment as analogy completion in a narrative context, rather than unrestricted narrative analogy discovery. The relevant source and target entities are pre-selected and the correct target keyword occurs in the supplied narrative; the experiment tests whether source relational information assists guided target completion. The reported results use stories with corrected source and target assignments.

  4. We agree that we cannot establish that these older benchmarks were absent from the models' pretraining corpora. Our previous observation of no exact overlap with the FFV-construction data addresses intervention-library leakage, not pretraining contamination. We now use “held-out” only to describe relations excluded from the FFV library. We also evaluated Top-5 accuracy on the target query alone (Table 18); no model above 12% suggests that the reported performance is not explained simply by predicting the answer from the target term. This control does not rule out exposure to complete benchmark items.

  5. We agree and now distinguish parameter efficiency from inference efficiency. AFV keeps the LLM frozen and requires no analogy-specific training, but the full pipeline takes about 5-19 s per analogy across the six models with initialization and composition alone taking 1-5 s. Source refinement is the major additional cost due to repeated forward-backward passes. Scaling to substantially larger relation libraries or models remains a practical limitation.

  6. In the stage-wise ablation (Table 26), target alignment changes accuracy by at most 1.1 pp on SAT/Turney for the six models. Its measurable effect is on self-consistency: it reduces KL(pT∣∣w)KL(p_T||w) by 6-38% (p<.001) while diverging from the source-derived distribution only modestly. Overall, the paper's central claim emphasizes source-conditioned composition; source refinement adds accuracy in some model regimes, while target alignment changes relational self-consistency rather than accuracy.

Questions

  1. Most of the gain comes from the source-conditioned composite (Weakness 1). The quality of the FFV’s library is also important: for GPT-J, replacing the FFV library with the initial-FV library reduces SAT accuracy from 50.27% to 37.43% and Turney accuracy from 29.41% to 24.60% (Table 24).

  2. See Weakness 2 and Appendix Table 16.

  3. We ran sensitivity analyses for injection layer and composition temperature. Appendix Figure 8 shows layer 12 to be among the strongest locations for the tested GPT and LLaMA models. For composition temperature, Table 20 reports τ∈{0.25,0.5,1,2}\tau \in \{0.25, 0.5, 1, 2\}. Performance is relatively stable for the LLaMA and Qwen models, whereas the GPT models can benefit from lower τ\tau; we nevertheless use τ=1\tau=1 uniformly. We did not run a sweep on the number of attention heads or the pruning threshold beyond wz<0.001w_z<0.001. This threshold removes approximately half of the relations while discarding only about 1% of the initial mixture weight, and our library-size ablations do not change the main conclusions.

  4. See Weakness 4.

  5. Our Limitations section states that extracting the relevant relational entities from paragraph-length narratives is a substantially harder problem and remains future work.

  6. Each model currently needs its own library, as the vectors live directly in its activation space. We have not tested cross-model transfer or learned mappings between models' relational spaces; this is for future work.

  7. The held-out type results for GPT-J are in Appendix Tables 22 and 23. The model improves on 9 of the 10 held-out SemEval relation types (e.g. +17.56 pp for Contrast, +17.11 pp Non-attribute). Class Inclusion is the exception with 54.55% for one-shot against 44.05% for the composite and 44.73% after source refinement. This pattern suggests that intervention is most useful when the baseline cannot already infer the relation, with examples in Table 29.

Thanks for the detailed response. The authors have addressed my concerns and I will increase my score.

Reviewer_mvJDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In this work the authors study whether compact activation-space representations can support analogical reasoning in language models. They first extract relation-specific function vectors through causal mediation analysis and then optimize these vectors with the language model frozen. They compose the resulting fine-tuned function vectors (FFVs) using weights determined by how well each relation explains a source word pair. Finally, they introduce Analogical Function Vectors (AFVs), which refine these weights by reconstructing the source answer and aligning the mixture with the model's analogy-conditioned target predictions. The experiments cover four models, from GPT-2-Medium to LLaMA-3.1-8B, and evaluate relation completion, agreement with human relational-similarity judgments, held-out relation types, four-term analogies and narrative-context analogies. The main results are substantial improvements in relation completion and consistent gains over one-shot prompting on analogy benchmarks. The evidence for the additional refinement stages, and for interpreting the learned vectors as the model's underlying relational mechanisms, is less conclusive.

Critical strengths
  • The question is important and well motivated. The paper goes beyond asking whether an LLM can answer an analogy correctly. It asks whether a reusable internal representation can carry a relation across domains. Connecting this question to function vectors gives the work a concrete intervention-based approach rather than relying only on behavioral correlations.
  • The learned vectors produce substantial gains. On the six relation-completion tasks in Table 1, FFVs raise GPT-J's average Top-1 accuracy from 56.75% with the initial FV to 87.01%, and LLaMA-2's from 68.45% to 88.91%. These are meaningful improvements, not small changes within an already saturated benchmark. The comparison with randomly initialized trained vectors in Figure 10 also provides useful evidence that the initialization helps.
  • The held-out-type evaluation is a strong part of the paper. Withholding an entire SemEval relation type is more informative than holding out new pairs from a known relation. The LOTO gains of 10.7, 10.4, 4.0 and 4.5 percentage points across the four models support transfer beyond the available relation labels. Treating relations, rather than the 40,770 generated analogies, as the unit of analysis is a thoughtful choice. The additional LTTO results strengthen this part of the evaluation.
  • The human-similarity analysis is more careful than a visual comparison of matrices. The authors bootstrap over word pairs, report confidence intervals and permutation tests, and control for same-relation membership. They also report that FFVs do not reliably improve over initial FVs in the first human experiment. This makes the result more credible and scopes it appropriately.
  • The framework separates reusable learning from per-problem adaptation. The base model remains frozen, and the analogy-time updates do not require the target answer. A reusable library with a small set of adaptable coefficients is a practical design. Its usefulness does not depend on proving that the model reasons exactly as humans do.
  • The appendix contains useful controls and candid qualifications. These include prompt-template changes, library-size ablations, random initialization, inference costs and the shared direction among FFVs. The authors also acknowledge that narrative evaluation uses pre-provided keywords. These details help distinguish what the method demonstrates from what remains open.
Critical weaknesses
  • The evidence supports composition more strongly than the full AFV refinement mechanism. This is my main concern about the central contribution. In Table 12, LLaMA-2's composite vector already achieves 70.00% on Green's far analogies and 48.40% on Turney's narratives; AFV refinement changes neither result. Its SAT improvement over the composite is only 0.54 percentage points. For LLaMA-3.1, refinement lowers far-analogy accuracy from 70.00% to 67.50% and adds only 0.27 points on SAT and narratives, approximately one additional correct item in each 374-item dataset. The GPT models benefit more, which is worth reporting, but these results do not establish that source-target alignment is generally necessary. The current tables also combine source refinement and target alignment, so they cannot attribute the benefit to either phase. I would like a direct comparison of initialization only, source refinement only, and both phases, with a measure of relational consistency as well as accuracy. Otherwise, the authors should frame composition as the main result and refinement as a model-dependent extension.

  • The related work misses several close connections. Hendel et al. (2023), In-Context Learning Creates Task Vectors: explain what relation-specific learning and held-out-type composition add to demonstration-derived task vectors. Liu et al. (2024), In-context Vectors: distinguish relational transfer from existing compositional activation steering, ideally with a simple vector-combination control. Wu et al. (2024), ReFT: Representation Finetuning for Language Models: discuss frozen-model representation learning and consider a matched-data, matched-parameter baseline for the efficiency claim.

  • The paper needs behavioral controls that distinguish relational composition from generic steering or selection. Appendix A.4.1 reports mean pairwise FFV cosines of 0.70-0.74 and notes that an uncentered composite is not reliably closer to the withheld relation's vector than the nearest single base vector. The mean-centered reconstruction analysis is useful, but recovering another trained vector is not the same as showing why predictions improve. A common direction could help the model produce relation-completion-style outputs, while the source-conditioned weights might add less than the framing suggests. Please compare against a norm-matched library mean or uniform mixture, the single vector with the highest source likelihood, shuffled composition weights, and an output-space mixture of the same relation-conditioned predictions. These controls would separate a shared task direction, relation selection, prediction ensembling and activation-space composition. The existing random-initialization experiment addresses how FFVs are learned; it does not resolve this question.

  • Successful interventions do not by themselves identify the model's native relational mechanism. Causal mediation analysis gives the initial vectors a meaningful origin, but the subsequent optimization freely changes the vectors to improve a supervised objective. Showing that the optimized intervention works establishes a useful way to control the model, not necessarily that the model originally represents or composes relations in this form. Similarly, readable decoded tokens, t-SNE clusters and agreement with human similarity judgments do not establish that the model uses these representations during ordinary analogy solving. The human-similarity controls are welcome, but the paper should also state how it selects the injected vector for each human stimulus and whether those word pairs were excluded from vector extraction and optimization. I would distinguish learned relational interventions from discovered native mechanisms. Stronger mechanistic claims would need evidence that the corresponding directions participate in unmodified inference, with selective ablation and recovery or another faithful causal test.

  • The alignment objective is a plausible self-consistency heuristic, not an independent check of relational correctness. In Section 3.3, the target evidence comes from the same model's predictions under the source-refined intervention. A base relation receives more weight when its target distribution agrees with those predictions. This can reinforce an initially wrong analogy, and agreement between output distributions does not ensure that the source relation is preserved. Moreover, for a fixed posterior p_T, Equation (3) has the direct optimum w = p_T on the retained support. Why use three optimizer steps instead of assigning those weights directly? If the posterior changes during those steps, the update rule and gradient handling need to be specified. I would like an ablation against the direct posterior update and examples showing when target alignment corrects an initially wrong prediction versus reinforces it. Claims that this step "enforces" relational consistency are stronger than the objective and current evidence support.

  • The supervision and compute budgets need clearer accounting. "As few as 10 relational examples" describes the additional optimization examples, not necessarily all relation-labeled data used to obtain the vector. The initial FV also uses separate examples, and SemEval initialization pools examples at the relation-type level. Appendix A.3.1 says the LoFiT FFV comparison uses the refinement pairs, so the authors should count the initialization data when comparing total supervision. The test-time comparison has a separate imbalance: AFV probes the base-vector library and performs an average of 29-109 source-refinement forward-backward passes per analogy, whereas the one-shot baseline performs no such adaptation. The reported timings are helpful, but parameter efficiency is not inference efficiency. A quality-versus-latency comparison, alongside same-data demonstration retrieval or explicit relation-inference prompting, would show what the intervention adds over simpler uses of the available information. I would not require every baseline to use identical computation, but the paper should make the trade-off explicit.

  • The analogy results need uncertainty estimates suited to their construction. The relation-level analysis for SemEval is good practice, but comparable analysis is missing from the main Green, SAT and narrative comparisons. Green's far-analogy set has only 40 problems, so a 12.5-point gain represents five items. Turney's 374 problems come from only 20 domain pairs and share generated narratives; they are not 374 independent narrative settings. Please report paired uncertainty estimates and resample at the domain-pair level for Turney. Averaging the base vectors over five seeds makes later evaluations deterministic conditional on that library, but it does not eliminate uncertainty from the sampled training pairs, the vector library or the evaluation set. This matters especially for the one- or two-item refinement gains on the larger models.

  • The evaluation supports a narrower reasoning claim than some of the discussion suggests. The authors clearly state that they convert SAT questions into generative completion by retaining the correct target pair. That is a legitimate task, but it is not the original five-choice SAT problem, and the paper should avoid treating the two as interchangeable. In the narrative task, the target keyword is present in the supplied context, and the relevant query terms are pre-selected. The experiment therefore tests guided completion within a narrative, not discovery of an analogical mapping from unrestricted text. A target-context-only baseline, a mismatched-source control, and distractor or counterfactual mappings would help establish that the gain depends on transferring the source relation rather than improving lexical retrieval. The limitations section acknowledges part of this already. Similarly, "out-of-distribution" should consistently mean held out from the FFV library, not absent from pretraining; no exact pair overlap does not establish that the pretrained model has never seen the relation or benchmark.

Questions for authors
  • How much does target alignment add beyond source refinement alone, and does it improve a measured notion of relational consistency when accuracy stays unchanged?
  • What are the total labeled-example and inference budgets for each method, including causal initialization, and how do the main comparisons change when those budgets are accounted for?
  • Do the small AFV-over-composite gains remain convincing under paired evaluation and domain-clustered uncertainty estimates?
1 comment

We thank reviewer mvJD for the constructive response.

Weaknesses

  1. We agree that composition is the primary source of the AFV’s accuracy gains. The stage-wise ablation (Table 26) shows that composition accounts for most of the improvement while source refinement adds gains mainly for the GPT models and LLaMA-2 on SAT. Source-target alignment encourages relational consistency with target-side evidence rather than maximizing performance: across the six models on SAT and Turney it reduces KL(p_T||w) by 6-38% (p<0.001) with little or no systematic cost in accuracy.

  2. Per the revised Related Work section: prior work induces or trains task-specific interventions from demonstrations, whereas we infer, compose, and refine a relational intervention for an unlabeled analogy from a reusable basis of relation vectors, even when its relation is not in the basis.

  3. We added the four controls in Appendix Table 17. First, a shared task direction is not sufficient: the composite outperforms the uniform and shuffled mixtures by 5.6-18.7 pp in most settings. Second, relation selection explains only part of the effect: for the LLaMA and Qwen models the composite matches or outperforms the single most likely finetuned function vector (FFV), while for GPT-J and GPT-2 on Turney the single relation outperforms the composite. The AFV narrows this gap for the GPT models (-2.9 to -0.8 pp GPT-J SAT, -7.5 to 0.0 pp GPT-J Turney, -8.3 to -1.9 pp GPT-2 Turney). Third, activation-space composition is not equivalent to prediction ensembling: against an output-space mixture the composite is better for both LLaMA models and slightly worse for the GPT models and Qwen2.5 in several cases. We now claim that the gains cannot be explained by a shared direction or arbitrary mixing; the composition’s advantage over selecting or ensembling relations is model-dependent.

  4. We now describe the FFVs as learned relational interventions rather than native relational mechanisms. When all human-similarity stimulus pairs are withheld from FFV optimization the correlation with human judgments shifts by at most 0.016.

  5. We agree that source-target alignment is a self-consistency objective rather than an independent measure of relational correctness and have removed any language that it “enforces” a relation. p_T is computed once from the source-refined weights and held fixed. In the direct-posterior ablation (Top-5 %) below, the 3-step KL update beats w = p_T in 7 of 8 comparisons (all but GPT-J on SAT).

ModelSAT: KL StepsDirect w = p_TTurney: KL StepsDirect w = p_T
GPT-232.632.434.834.5
GPT-J50.351.629.428.1
LLaMA-260.757.262.661.0
LLaMA-3.162.360.763.461.5
  1. Each initial FV is extracted from two sets of 25 10-shot prompts, at most 500 demonstration pairs per relation (550 with queries). FFV optimization then uses at most 10 examples per relation for a total of 2,356 pairs. For relation completion LoFiT is trained on the same pairs used to optimize the FFVs, while for analogies it is trained on 2,898 four-term problems constructed from at most 10 of those pairs per relation, whereas AFV receives no analogy training and inherits information from the FV extraction. We also agree that parameter efficiency should not be conflated with inference efficiency: the AFV keeps the LLM frozen but takes 5.3-18.8 s per analogy (Table 8). Our claim is parameter efficiency and absence of analogy-specific training, not lower inference cost than prompting.

  2. We added paired 95% bootstrap confidence intervals for AFV minus the one-shot baseline (Table 27), resampling the analogies for Green and SAT and the 20 domain pairs for Turney. The CIs exclude zero for five of six models on SAT and for GPT-2 and Qwen2.5 on Turney; the smaller Turney gains are not distinguishable from zero. Green’s far-analogy gains are also positive on all models. Across five independently sampled FFV libraries the composite’s accuracy varies little (e.g., SDs of 0.2-1.1 pp for GPT-J, 1.4-2.3 pp LLaMA-3.1).

  3. We now focus on analogy completion and relational transfer rather than unrestricted analogy discovery: SAT uses only the correct answer as the target rather than the five-choice task, while Turney tests guided target completion with pre-selected queries and the target keyword present in the narrative context. We also replace “out-of-distribution” with “held-out” terms for relations excluded from the FFV library.

Questions

  1. Target alignment provides little additional accuracy beyond source refinement. We discuss this in Weakness 1.

  2. See Weakness 6.

  3. Not uniformly; composition accounts for most of the improvement. On Turney, AFV minus composite yields +6.4 pp [95% CI +3.1, +9.5] for GPT-2 and +7.5 pp [+3.9, +11.1] for GPT-J. In contrast, the differences of several larger models are at most +1.6 pp, so we frame refinement as a model-dependent extension.

Comments on the paper

1 comment

We thank the reviewers 1iU6 and iYyi for their responses during this discussion period. We did not receive comments from reviewer mvJD. As the discussion period concludes, we summarize our main updates in the revised manuscript in response to the reviewers:

  1. Clarification of the roles of AFV composition, source refinement and source-target alignment (Reviewers mvJD, 1iU6, iYyi). The results of our stage-wise ablation (Table 26) show that composition accounts for most of the accuracy improvement, with source refinement adding gains mainly for the GPT models and LLaMA-2. Source-target alignment serves a different role: rather than directly maximizing answer accuracy, it encourages agreement with target-side relational evidence. Across six models on SAT and Turney, alignment reduces KL(p_T||w) by 6–38% (p<0.001) with little or no systematic change in accuracy. We revised the paper’s framing to distinguish these contributions more clearly: composition is the primary mechanism for held-out-relation generalization; source refinement is a model-dependent accuracy-improving extension; and source-target alignment is an optional self-consistency refinement rather than an independent guarantee of relational correctness.

  2. AFV goes beyond generic steering and selection (Reviewer mvJD). We added four controls (Table 17) to determine whether the observed gains could be explained by simpler mechanisms. First, a shared task direction is not sufficient: the composite outperforms the uniform and shuffled mixtures by 5.6-18.7 pp in most settings. Second, relation selection explains only part of the effect: for the LLaMA and Qwen models, the composite matches or outperforms the single most likely FFV. Third, we compared activation-space composition with an output-space mixture using the same relation weights. The composite outperforms this control for both LLaMA models, while the comparison is mixed for the GPT and Qwen models. We now claim that the gains cannot be explained by a shared direction or arbitrary mixing, while the composition’s advantage over selecting or ensembling relations is model-dependent.

  3. Expanding evaluation to newer model families and Top-1 results (Reviewers 1iU6 and iYyi). We added all major evaluations for the Qwen family (Qwen2.5-7B-Instruct, Qwen3-8B) in the main text, with their tables and the SmolLM family (SmolLM2-1.7B, SmolLM3-3B) evaluations in the appendix. For the Qwen models, the composite and AFV improve one-shot accuracy on Green far-analogy, SAT, and Turney, though the smallest gains are from Qwen3-8B which has the strongest baseline (55.9% SAT). For the SmolLM models, the composite improves over the one-shot baseline across all benchmarks, especially on the Green far-analogy and SAT datasets. We also added Top-1 accuracy in Table 16: the AFV exceeds the one-shot baseline across all six models on the Green far-analogy, SAT, and Turney datasets, though the refinement's improvements over the composite are model-dependent. Top-5 accuracy remains the primary metric as the generative task can admit lexical competition.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.