acceptodds
Under review as a conference paper at ICLR 2027

Learning Transferable Directions

Abstract

Transferability need not be a property of an entire source model. A source update that is globally misaligned with a target can contain locally useful computations, while a globally similar source can move particular target inputs in a harmful direction. We introduce Learning Transferable Directions (LTD), which represents each reusable source block by the exact function-space change it induces at a target input and learns a target loss-descent direction field from cross-fitted target gradients. A cheap proposal network first shortlists source blocks; only those blocks are evaluated, after which a sparse nonnegative quadratic program composes locally aligned directions. The construction requires source checkpoints but not source examples and nests target-only prediction exactly. We establish a smoothness-based directional-gain bound, exhibit a partial-sharing setting in which every input-independent source-weight rule incurs a nonvanishing gap, and give an optional finite-sample validation gate that falls back to target-only prediction unless transfer is certified nonharmful for a prespecified bounded loss. Simulations and real-data experiments show that LTD improves target performance in low-resource regimes, reduces negative transfer, and achieves favorable performance-computation tradeoffs over model-merging and routing baselines across vision and language benchmarks.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Function-space alignment, Input-dependent routing, Model merging, Negative transfer, Transfer learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_1pRJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Learning Transferable Directions (LTD), which performs transfer at the level of input-dependent functional changes induced by source blocks. LTD estimates a target descent direction, shortlists candidate source blocks, and combines their functional directions through sparse optimization. Experiments cover vision, domain shift, and language settings, with generally positive results.

Critical strengths

The idea of measuring transferability through local functional effects rather than global parameter similarity is interesting. The experimental evaluation is also reasonably broad and includes useful ablations.

Critical weaknesses

The main motivation is not sufficiently clear. In particular, I found it difficult to understand why LTD is fundamentally preferable to simpler input-dependent routing over source components. The paper introduces substantial machinery around a relatively simple intuition, including direction-field estimation, proposal learning, sparse optimization, and several selection stages, without convincingly justifying this complexity. More generally, the presentation is difficult to follow and the distinction from related routing and expert-combination approaches remains unclear.

Questions for authors
  1. Can the authors provide a simple setting where standard input-dependent routing fails but LTD succeeds specifically because of functional-direction alignment?

  2. How sensitive is LTD to the choice of source-block partition and fusion layer?

  3. What is the full target-adaptation computational cost, including direction computation, cross-fitting, proposal training, and model selection?

Reviewer_bwdKOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LTD, which treats transferability as a local relation between a reusable source computation’s functional change and the target loss-descent direction at a given input. It develops a checkpoint-only transfer algorithm that requires the base and source checkpoints but no source examples; a cheap proposal stage shortlists candidate blocks, exact functional directions are evaluated only for that shortlist, and a sparse nonnegative quadratic program composes the resulting correction. The method also includes an optional target-only safety gate. The theory characterizes approximation to a local directional oracle, the limits of input-independent source weights, and finite-sample fallback protection. Experiments evaluate "predictive performance and the proposed geometric mechanism through low-resource adaptation, model-merging benchmarks, negative-transfer diagnostics, and interventions on selected source directions."

Critical strengths
  1. The paper clearly distinguishes local functional transfer from global parameter or functional angles. It "defines a transferable direction as an input-dependent functional change" and shows that "in linear models, global parameter angles arise as a second-moment-weighted special case." The separation result also isolates why "every input-independent source-weight rule" can average away useful region-specific information.

  2. The algorithm separates proposal screening from exact directional evaluation and nests target-only prediction. The safety result is appropriately scoped: it "controls only the prespecified bounded risk" rather than claiming pointwise nonharmfulness or protection for every downstream metric.

Critical weaknesses
  1. The method relies on a prespecified fusion layer and block partition, and it "assumes that the base and source checkpoints share a parameterization." The main text does not provide systematic evidence on sensitivity to these choices, and "richer functional dictionaries and heterogeneous source architectures remain for future work."

  2. LTD-SAFE certifies only a task-level bounded validation risk. It "does not claim pointwise nonharmfulness or protection for every downstream metric", and ordinary LTD selection "does not by itself rule out choosing a transfer configuration whose population risk exceeds that of target-only prediction".

Questions for authors
  1. How sensitive are the reported gains to the choice of fusion layer and source-block partition, and what practical guidance does the method offer when the architecture lacks a natural modular partition or when source checkpoints do not share a parameterization?

  2. Since LTD-SAFE’s certificate is global and tied to a prespecified bounded loss, how should practitioners interpret a certified model when the deployment metric is ordinary cross-entropy or accuracy, or when individual inputs may still be harmed?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.