Learning Transferable Directions

Transferability need not be a property of an entire source model. A source update that is globally misaligned with a target can contain locally useful computations, while a globally similar source can move particular target inputs in a harmful direction. We introduce Learning Transferable Directions (LTD), which represents each reusable source block by the exact function-space change it induces at a target input and learns a target loss-descent direction field from cross-fitted target gradients. A cheap proposal network first shortlists source blocks; only those blocks are evaluated, after which a sparse nonnegative quadratic program composes locally aligned directions. The construction requires source checkpoints but not source examples and nests target-only prediction exactly. We establish a smoothness-based directional-gain bound, exhibit a partial-sharing setting in which every input-independent source-weight rule incurs a nonvanishing gap, and give an optional finite-sample validation gate that falls back to target-only prediction unless transfer is certified nonharmful for a prespecified bounded loss. Simulations and real-data experiments show that LTD improves target performance in low-resource regimes, reduces negative transfer, and achieves favorable performance-computation tradeoffs over model-merging and routing baselines across vision and language benchmarks.