acceptodds
Under review as a conference paper at ICLR 2027

Cayley-Parameterized Task-Vector Transfer: Invariant Re-basing of Fine-tuning Updates

Abstract

Task vectors capture the changes in a model's weights from fine-tuning and offer a way to reuse learned knowledge across models. Effective task-vector transfer, however, depends on how these updates are aligned and scaled to the recipient model. We introduce Cayley-Parameterized Task-Vector Transfer (CTT), which jointly learns orthogonal residual-stream rotations and scaling factors that re-base the task vectors from the original model to the recipient model. We parameterize the rotations with the Cayley transform, which keeps them orthogonal and differentiable throughout training. We evaluate CTT through three sets of experiments. First, we compare it against other state-of-the-art task-vector transfer methods on both image and text domains using CLIP-ViT B/16. On average, CTT recovers more of the domain than the other methods on the text domains, and performs comparably well to the other methods on the image domains. Then, we apply CTT to natural language autoencoders (NLAs), which describe model activations in natural language and reconstruct them from those descriptions. Since NLAs degrade when their underlying base models are fine-tuned and struggle to reconstruct out-of-distribution activations, task-vector transfer provides an effective way to restore them without full-parameter retraining. We find that CTT consistently recovers more of the NLA domains than the other methods. Finally, we compare CTT directly with GRPO and SFT fine-tuning of the NLA. Under the same training-data distribution, CTT achieves comparable code-domain recovery and higher support retention while using significantly fewer estimated training FLOPs than GRPO, and achieves greater code-domain recovery than SFT. Together, these results show that CTT is the only method among those we compare that respects reparameterization symmetries and transfers task vectors effectively across models. The differences that we observe between the image and text domains also suggest that the effectiveness of task-vector transfer methods may be modality-specific.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
task vectors, task-vector transfer, linear mode connectivity, language modeling, deep learning, transfer learning, natural language autoencoders, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fNLxOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

In the work the authors introduce Cayley-Parameterized Task-Vector Transfer (CTT), a method for transferring fine-tuning updates across pretrained models with the same architecture by learning orthogonal residual-stream rotations jointly with their scaling coefficients. The Cayley transform is used to maintain orthogonality diring optimization, with the motivation that the resulting transformation respects reparameterization symmetries and thus changes the basis of transferred update rather than its content. The method is evaluated on CLIP image and text tasks and on natural-language autoencoders.

Critical strengths

CTT has a clean motivation, the transferred task vector is transformed only through an architectural reparameterization symmetry, with the Cayley parameterization maintaining orthogonality throughout optimization.

The paper evaluates the method on CLIP image/text transfer and on natural-language autoencoders, and compares against several recent task-vector transfer methods.

Critical weaknesses
  1. The experimental protocol is not sufficiently described in the main paper. Essential information needed to interpret the results are buried in the very long supplementary material without any pointer. For example, the definitions of TASK/REC/SUPP/RET are missing and are different from usual settings, the most similar is TransFusion which reports TASK and SUPP. The few-shot budget, optimization details and validation protocol are all hidden in the appendix. The most critial is the averaging procedure of all budgets at different seeds which should be clearly disclosed in the main paper. All these lack of details make it impossibile to interpret the results without reading the 30+ pages appendix.

  2. The CLIP experiments do not isolate transfer from supervised adaptation. CTT is not a label-free transport procedure. It takes the labeled examples from the target task and explicitly optimize R and α\alpha by cross-entropy on those labels for 500 updates (this is unclear, but seems the case from the appendix). At this point the question is obvious: why not simply adapt the target model using such samples. Without this comparison it is hard to know if CTT is exploiting knowledge in the source task vector or simply is a constrained parameterization for few shot supervised adaptation. From the original CLIP paper we know that 16 shots is already more than enough to fit the model on the downstream task. Note that GradFix and BiCo make use of the gradients, but do not directly use gradient descent, moreover, GradFix reports a same budget target finetuning comparison.

  3. Missing baseline. CTT is initialized from a validation-selected task vector scale, but this tuned naive-transfer baseline is not reported. A tuned-scale or α\alpha-only baseline is needed to separate the gain from scale selection from the gain due to the Cayley rotation.

  4. The comparison across transfer methods is not budget-matched. Different methods use substantially different amounts and types of information: some are data-free, some use unlabeled data, while CTT uses labeled examples, gradient-based optimization, and repeated validation-based checkpoint selection. These differences are mostly hidden in the supplement and complicate claims of superiority.

  5. The empirical evidence for generality is limited. The CLIP experiments consider a single source/recipient checkpoint pair with the same ViT-B/16 architecture, and support retention is evaluated rather narrowly. This seems limited relative to the broader framing of task-vector transfer across pretrained models.

Questions for authors
  1. Can you compare CTT against direct adaptation of the recipient CLIP model using exactly the same labeled examples and a matched optimization budget, including full fine-tuning and/or LoRA?

  2. Can you report the validation-tuned naive-transfer baseline used to initialize CTT, together with an α\alpha-only learned baseline, so that the contribution of the Cayley rotation can be isolated?

  3. Can you provide a matched accounting of the labeled/unlabeled data, optimization steps or FLOPs, and number of validation evaluations available to each method?

  4. Can the essential experimental protocol currently buried in the supplement, including the definitions of TASK/REC/SUPP/RET, the KK-shot budgets and averaging procedure, checkpoint pair, and model-selection protocol, be moved into the main paper? In the current version, I do not think the central tables are self-contained enough to assess the claims without substantial appendix reading.

Reviewer_tWTSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CTT adapts a source model’s fine-tuning update to a structurally matched target model by learning a shared rotation of its residual-stream coordinates and a scaling factor. The paper proves that this transformation respects reparameterization symmetry under stated architectural conditions; transfer quality remains an empirical question. Results are strongest on CLIP text tasks and NLA reconstruction, while several CLIP image tasks show a clear trade-off between recovering the new skill and retaining existing performance.

Critical strengths
  1. CTT gives task-vector transfer a coherent geometric constraint: one residual-stream rotation is applied consistently across layers, and the paper formally analyzes when this operation respects reparameterization symmetry. This distinguishes it from methods that fit separate transformations at each layer.

  2. The evaluation spans CLIP image and text encoders and two NLA backbones. The method performs particularly well on CLIP text tasks and NLA reconstruction, while the appendix documents data splits, baseline adaptations, and evaluation procedures in useful detail.

Critical weaknesses
  1. The central mechanism is not isolated. The paper does not compare learned rotation alone, learned scale alone, and their combination under a matched fitting and selection budget. This is especially important because CLIP runs use a validation-tuned initial scale.

  2. Comparisons with training-free alignment methods conflate alignment quality with access to task data, optimization, and checkpoint selection. Matched resource comparisons and recovery–retention curves would make the practical advantage clearer.

  3. The theoretical guarantee concerns a particular function-preserving parameterization and the set of candidate transfers; it does not guarantee the behavior of a finite optimization run. The paper provides numerical consistency checks, but an inspectable implementation would be needed to audit every required transformation.

  4. The empirical scope remains narrow in two ways: cross-width and cross-depth transfer are untested, and direct fine-tuning of the CLIP recipient is absent. CLIP image retention losses and small NLA margins over the strongest baseline also warrant a more cautious overall claim.

Questions for authors
  1. Could you compare three variants under the same fitting budget: learned rotation with a fixed scale, learned scale with the identity rotation, and the full method? Reporting both recovery and retention would help establish the contribution of each component.

  2. What accounts for the ImageNet-R retention losses on DTD and EuroSAT? In particular, could selecting checkpoints by target-task accuracy contribute to these losses? A recovery–retention curve would help show the available trade-off.

  3. What practical advantage does transferring a source task vector provide over directly adapting the recipient with LoRA? A comparison using the same task data, with trainable parameter counts and training FLOPs reported, would make this distinction clearer.

Reviewer_WiNDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies task-vector transfer across pre-trained models. The authors argue that existing methods struggle to simultaneously achieve effective transfer and respect reparameterization symmetries. To this end, they propose CTT: jointly learning an orthogonal rotation (parameterized by the Cayley transform) and a scaling coefficient on the residual stream, so that transfer only changes the coordinates of the task vector. Experiments cover CLIP and natural language autoencoders (NLA). The authors report that CTT performs best among symmetry-respecting methods and, on NLA, achieves comparable performance at a cost far lower than direct training.

Critical strengths
  1. This work tackles a practical question: how fine-tuning updates can be reused across successive pretrained models.

  2. The proposed approach is both simple and computationally efficient.

  3. The evaluation covers image, text, and NLA settings, while the appendix documents the protocols in sufficient detail for replication.

Critical weaknesses
  1. There are serious problems with the writing. The current version of the Abstract does not meet the ICLR acceptance standard; it includes many details that should not be elaborated in an abstract.

  2. There is no experimental isolation of the source of gains. There is no setting that learns only scaling, only rotation, a naive baseline with tuned alpha, or a random task-vector control. On NLA, the scaling learned by CTT reaches 3–8×, whereas the naive and permutation baselines fix alpha=1.

  3. The main conclusions are overstated. (1) The text-side claim of “recovering more on average” relies on degenerate RTE cells; after removing RTE, CTT is lower than BiCo. (2) The main table does not report variance, and on NLA 4B, CTT’s lead over Kawamoto is smaller than the seed-to-seed gap reported in the paper itself. (3) On the image side, it underperforms BiCo for every K, and the retention rate also drops.

Questions for authors

see weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.