ReFract: Model Re-basin through Decomposable Fine-Tuning Residuals

Model re-basin enables the transfer of task-specific capabilities between models, mitigating the challenges of full retraining. However, existing parameter-space approaches typically rely on weight correspondences that are difficult to establish across different architectures. To address this limitation, we shift the focus to activation space and transfer the functional residual, i.e., the change in last-layer hidden representations induced by fine-tuning. Under standard non-linear fine-tuning, this residual entangles the contributions of individual parameter updates through non-linear interactions, making them difficult to isolate and transport to another model. We show that, when the donor model is fine-tuned in the neural tangent kernel (NTK) regime, the residual admits an exact additive decomposition over parameters, without higher-order interactions. Like a prism separating light into its constituent components, we isolate the functional contribution of each layer and transport it to corresponding layers in another model. In practice, our approach first transports these activation residuals into the target model's feature space and then learns layer-wise predictors that map its frozen intermediate activations to the transported corrections. We achieve state-of-the-art performance in capability transfer across both vision and language models, approaching full-data target fine-tuning using only a few examples from the original dataset.