CORE-FV: FROM MARGINAL TO CONDITIONAL HEAD SELECTION FOR FUNCTION VECTORS

Function vectors provide a compact mechanism for extracting and transferring task-specific behaviors from in-context demonstrations by aggregating the contributions of a small subset of attention heads. Existing approaches typically identify these heads using marginal importance scores, but evaluating heads independently overlooks interactions among them, causing redundant or complementary heads to be selected suboptimally. We introduce CORE-FV (Conditional Ordering through Removal Evaluation for Function Vectors), a conditional head selection framework that evaluates each attention head in the context of the remaining intervention set. CORE-FV performs an adaptive backward refinement procedure that iteratively removes heads whose absence minimally affects the induced task behavior, producing an interaction-aware ordering of heads. Across eight in-context learning transformations and multiple open-weight language models, CORE-FV generally outperforms a matched Average Indirect Effect baseline, achieving average accuracy gains of up to 11.84 percentage points. Moreover, on composite tasks requiring sequential or parallel transformations, CORE-FV substantially outperforms marginal head selection, improving average accuracy from 41.81% to 87.34% on Llama-3.1-8B. These results highlight that head contributions can be inherently context-dependent, and that interaction-aware selection provides a more effective foundation for constructing compact and algebraically composable task representations.