Communicating Learning Directions Through a Shared Foundation Model

Adapting a large target model to private client data typically requires the client to receive and optimize that model. We study a target-decoupled setting where the client instead has access to a smaller shared reference model, while the server holds the target model and a PUBLIC repository. We introduce GradHop, which uses the private data to form learning directions under the reference model and communicates them once to the server. The server uses these directions to retrieve client-relevant examples from the PUBLIC repository and then adapts the target model directly on the selected examples. This avoids requiring parameter-wise correspondence between the reference and target models. We analyze this cross-model re-grounding and provide conditions under which reference-space retrieval identifies PUBLIC examples whose target gradient approximates the inaccessible private target gradient. Experiments on Banking77 and CLINC150 show improved target adaptation over the unadapted model and alternatives that use only PUBLIC data, while the target-space analysis confirms stronger alignment with the private target gradient. The same reference-side information also supports adaptation across different target families without exposing the target model to the client.