acceptodds
Under review as a conference paper at ICLR 2027

The Geometry of Transfer

Abstract

A new target task may have little labeled data but access to a library of fine-tuned models whose source datasets are unavailable. Existing transferability scores rank source representations, while model-merging methods combine task vectors, but neither directly quantifies which part of target adaptation is already contained in the source library. We formulate transfer as projection in a target-conditioned functional tangent space. Each source checkpoint induces a tangent field on target inputs, while the target loss induces a curvature-normalized descent field. Their weighted geometry defines a principal transfer angle for each source subspace and a transfer completeness index measuring the fraction of target adaptation captured by a sparse source span. We introduce GEOTRANSFER, which estimates this geometry from source checkpoints and a small target sample, performs group-sparse source projection, learns an orthogonality-regularized innovation adapter for residual target structure, and uses a held-out risk gate for target-only fallback. We prove a local projection identity, first-order invariance to smooth reparameterization, a sparse-span oracle inequality, and a finite-sample guarantee that gated deployment does not exceed the target-only baseline under a prespecified bounded target risk. Simulations and real-data experiments show that the geometry predicts realized source utility, is stable under parameter rescaling, and improves few-shot transfer when useful and harmful source directions coexist.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Function-space geometry, Model merging, Task vectors, Transfer learning, Transferability estimation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WDJ5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem of transferring knowledge to a target task with limited labeled data, given only a set of source checkpoints or adapters fine-tuned from a common pre-trained model, without access to the original source data. To define transfer in the function space of target inputs rather than in parameter space, the authors map each source update to a change in target logits via the base model’s Jacobian and compare it against a descent direction adjusted by the curvature of the target loss. Based on this, they define the principal transfer angle, which represents the utility of each source, and transfer completeness, which indicates the extent to which the source library covers the target adaptation. The proposed method, GeoTransfer, determines combination coefficients by solving an orthogonal projection problem with sparse-group regularization over layer-grouped source updates, learns the components unexplained by the sources using an adapter penalized for overlap with the selected source directions, and employs a risk gate that permits deployment only if a statistical test on validation data confirms that performance is not worse than a target-only model. Theoretically, the paper establishes projection optimality under a second-order approximation, first-order invariance to re-parameterization, an upper bound on the sparse estimation error that depends on the number of active sources, and a finite-sample guarantee for the risk gate. Experiments spanning controlled simulations, CLIP task libraries, cross-domain vision libraries, and language adapter libraries report better source ranking and few-shot performance than the baselines, along with a substantial reduction in negative transfer.

Critical strengths

Principled formulation and interpretable diagnostics. By defining source utility in function space rather than parameter space, the authors provide a criterion that is robust to parameter symmetries such as rescaling, and connect it to a clear geometric structure based on orthogonal projection. In particular, the completeness metric provides information that existing transferability scores or merging methods do not offer, since it quantifies not only the ranking of individual sources but also how much of the target adaptation the entire library can cover. The theoretical framework is also rigorous, with explicit assumptions and proofs, and a strict separation of the fit, validation, and test splits.

Practical safeguards and computational efficiency. The method includes a validation-based risk gate intended to control negative transfer, backed by a finite-sample guarantee. Furthermore, the required quantities are computed with Jacobian-vector products and a single backward pass, without constructing a parameter-dimensional Hessian, and at deployment the model runs at the same cost as a standard model, with no additional source models or routers.

Critical weaknesses

The positioning relative to related work and the choice of baselines are inadequate. The setting of adapting to a new target in a few-shot manner by combining task vectors fine-tuned from a pre-trained CLIP model with block-wise coefficients was already studied in aTLAS (Zhang et al., NeurIPS 2024) under nearly the same protocol (leave-one-out task vectors, k shots per class), yet this paper neither cites nor compares against it. There is also no comparison with LoraHub, which directly overlaps with the language experiments, or with AXIS (Osial et al., 2025), which the paper itself cites as the closest related work. In addition, Ortiz-Jiménez et al. (NeurIPS 2023), which studies task vectors in the tangent space and is closely related to the first-order view adopted here, is not cited or discussed. The current baselines consist mainly of methods originally designed for multi-task merging (e.g., TIES, TATR, TSV-Merge), which make little use of target labels, together with author-constructed baselines, and standard CLIP few-shot baselines such as Tip-Adapter and LP++ are absent. As a result, the remaining core contribution, namely determining coefficients through orthogonal projection in function space, is compared against direct optimization of coefficients on the target loss only through an author-constructed baseline whose granularity and regularization are not specified, and not against established methods such as aTLAS.

The empirical evidence on real data is weak. Table 2 does not report standard errors, the number of random seeds, or per-dataset results. Information needed for reproducibility is also missing, including the backbone, the number of LoRA experts, and the target tasks in the language experiments, as well as the block partitioning criterion in the vision experiments. In addition, the differences between methods in Table 2 are unusually constant across six very different settings; for example, applying the gate lowers performance by exactly 0.2 percentage points in every setting. The accuracy of the first-order (tangent) approximation, on which the method relies, is not measured on real models, even though real CLIP task vectors can lie far from the base model. Meanwhile, the simulation favors the proposed method, since the sources are generated by orthogonalizing latent factors under the same function-space inner product that the method uses. The source-ranking criterion, the gain from adding a single source at its optimal scale, is essentially the quantity that the proposed angle approximates, which makes the comparison with methods such as LogME questionable. Finally, the claim that the completeness metric predicts actual transfer gains is supported only by simulation results.

Questions for authors
  • Could you provide comparisons with aTLAS, LoraHub, and AXIS under the same conditions? In addition, please specify whether the Sparse target-loss weights baseline operates at the source level or the block level, and whether it uses the same atom partitioning as the proposed method.
  • Could you report the number of seeds, per-seed results with standard errors, and per-dataset results for Table 2 and Table F.3? Please also specify the backbone, the LoRA expert configuration, and the target tasks for the language experiments, as well as the criterion for the three-group block partitioning in the vision experiments.
Reviewer_PMyfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies the problem of transfer learning where we have no access to the source data but only have a library of models and some labels. It proposes a method called "GeoTransfer" that estimates the geometry that quantifies which part of target adaptation is already contained in the source library. The method uses the induced perturbation ϕs(x)\phi_{s}(x) of each fitted source and project it onto the desent field r0(x)r_0(x), which describes the geometric meaning of transferability. A completeness score is constructed for each source, then the method combines the source in a sparse way if the source pass the final safety check (i.e., the gate).

Critical strengths
  1. To my best knowledge, the proofs of the theorems are correct.
  2. The setup that we have access to model library rather than source data is of interest and importance.
Critical weaknesses
  1. There is a gap between the reported safety gate acceptance rate vs the method. By Eq. (3.9), the saftey gate accept a transferred model if its performance exceeds at least by 14B log⁡(2⋅∣Λ∣/δ)3 (nval−1)\frac{14B\,\log(2\cdot |\Lambda|/\delta)}{3\,(n_{\mathrm{val}}-1)}. This is inconsistent with the reported accept rate. For example, using B=1 (line 1614 in the paper), ∣Λ∣=972|\Lambda|=972, δ=0.05\delta=0.05, if nval=100n_val= 100, we have 14B log⁡(2⋅∣Λ∣/δ)3 (nval−1)\frac{14B\,\log(2\cdot |\Lambda|/\delta)}{3\,(n_{\mathrm{val}}-1)} equals approximately 0.216. But Table C.6 reports that candidate acceptance rate is 0.64 and mean gain when accepted is 0.032. The number 0.032 is significantly smaller than 0.216. The same problem applies to Table F.3. Am I using the wrong example of nvaln_val here? Can the authors explain why?

  2. In Table C.5, the functional rank \pho\pho and the support F1 changes when using different normalization. If the normalizing constant is identical to every source (i.e., the denumerator has the same number during normalization), how will the rank change? The ranking should keep unchanged, right? Can the authors explain?

  3. Table F.5 has show that the completeness score changes when having 25% sign-reversed or 50% sign-reversed. This is inconsistent with the claim in the paper (line 179, principle angles and χq\chi_q are orientation free by construction).

  4. Table C.3 shows non-zero functional drift to linear rescaling of parameters. The authors explains "finite endpoint linearization". It does not persuade me. A linear map should have the error term in Proposition 4.2 as 0, right? The second order Taylor's remainder term should be zero for any linear rescaling function.

  5. Theorems in the paper apply to a theoretical model, rahter than the model used by the author. The gap E^nl\hat{E}_{nl} (line 1468) needs to be handled.

  6. The assumptions of Theorem 4.3 are strong. The sparsity requires a larger sample size than shown in the experiments. Have the authors considered including experiments with more data points?

Questions for authors

Please address the above 6 comments in the weaknesses.

Reviewer_GNhaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates few-shot transfer that tries to untilize the information provided by a source-model library. It introduces target-conditioned functional geometry as a criterion to measure source usefulness, combing sparse updates with a target-specific adapter.

Critical strengths

The problem is meaningful: what can the source library jointly provide, and what remains to be learned?

The functional geometry gives a clear rationale for handling source alignment, redundancy, and residual adaptation.

The local theory and experiments provide useful support for the approach.

Critical weaknesses

The comparison with closely related task-vector adaptation methods, like Osial et al. (2025), could be clearer. More direct comparisons would help establish the practical benefit of the proposed geometry. Some experimental settings also need clarification, as noted in the questions below.

[1] Osial, M., Wojcik, B., Zieli ´ nski, B., and Cygert, S. (2025). Efficient multi-source knowledge transfer by model merging. arXiv:2508.19353

Questions for authors
  1. Could the authors clarify the sign-reversal experiment in Table F.5? My understanding is that, with signed coefficients, reversing source updates preserves their span. Were any additional source-filtering rules used, and were repetitions paired across conditions? This would help me understand the decline in completeness.

  2. Could the authors clarify how the normalization choices in Table C.5 affect support F1? My understanding is that support selection uses unnormalized projection energies. Did other estimation steps also change, or were the rows evaluated using independent Monte Carlo repetitions?

Overall

I score it as 3. The paper offers a useful perspective with a reasonable design and supporting evidence. The questions above seek clarification rather than identify established errors. By the way, I am not very familiar with this specific topic and have not checked the proofs in detail, so I assign a low confidence.

1 comment
Reviewer_GNhaOpenReview ↗

Final Assessment

Thank you for your response. I still see the paper as borderline between 2 and 3, and I will keep my rating at 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.