acceptodds
Withdrawn from ICLR 2027

Communicating Learning Directions Through a Shared Foundation Model

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Adapting a large target model to private client data typically requires the client to receive and optimize that model. We study a target-decoupled setting where the client instead has access to a smaller shared reference model, while the server holds the target model and a PUBLIC repository. We introduce GradHop, which uses the private data to form learning directions under the reference model and communicates them once to the server. The server uses these directions to retrieve client-relevant examples from the PUBLIC repository and then adapts the target model directly on the selected examples. This avoids requiring parameter-wise correspondence between the reference and target models. We analyze this cross-model re-grounding and provide conditions under which reference-space retrieval identifies PUBLIC examples whose target gradient approximates the inaccessible private target gradient. Experiments on Banking77 and CLINC150 show improved target adaptation over the unadapted model and alternatives that use only PUBLIC data, while the target-space analysis confirms stronger alignment with the private target gradient. The same reference-side information also supports adaptation across different target families without exposing the target model to the client.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Foundation models, one-shot federated learning, model adaptation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Wss1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GRADHOP to adapt a server-held target model to private client data without requiring clients to access or optimize that model. Each client uses a smaller shared reference model to compute learning directions and sends them to the server once. The server uses these directions to select examples from a PUBLIC repository and adapts the target model on the selected examples. The paper analyzes when this cross-model selection can approximate the private target gradient and evaluates the method on Banking77 and CLINC150.

Critical strengths
  1. Clients do not need to host or differentiate through the large target model, and they communicate with the server only once. This addresses a practical constraint when clients have limited computational resources or cannot access the target model.

  2. The reference and target models need not have matching parameters or gradient dimensions. Selecting PUBLIC examples in the reference space and optimizing them in the target space provides a clear way to connect two independently parameterized models.

  3. The experiments include two datasets, a target-gradient alignment analysis, an ablation of mode-specific directions, and tests across several target-model families. These results provide useful evidence for both the adaptation performance and the proposed selection mechanism.

Critical weaknesses
  1. The paper discusses related approaches such as FedProxy, Offsite-Tuning, and LESS, but the main comparisons are limited largely to random, label-balanced, and all-PUBLIC selection. Although these approaches have different access assumptions, the authors should explain the differences more carefully and compare with stronger methods under matched constraints. In particular, a reference-embedding or prototype-based PUBLIC retrieval baseline would help establish whether gradient directions are necessary.

  2. The main experiments consider one private client per adaptation run, while the multiple-client experiment reports only aggregate results. Since different clients may benefit differently from the adapted target, the authors should report per-client performance, worst-client performance, and the number of clients experiencing negative transfer. Average accuracy and NLL may conceal degradation for individual clients.

  3. The PUBLIC repository contains examples from the same set of intent labels used in each client task, and the evaluation is limited to intent classification on two datasets. The authors should test what happens when relevant PUBLIC examples are scarce or absent, when the PUBLIC and private distributions differ substantially, and when the method is applied to other tasks. The construction and size of the PUBLIC repository, as well as its separation from private and test data, should also be stated more clearly.

  4. Eq. (11) defines a positive reference-space margin by requiring the target-space oracle support to rank ahead of all other PUBLIC examples. This assumes a strong relationship between the two model spaces, but the paper does not establish when that relationship holds. Moreover, Eq. (13) concerns a single top-BB ranking, whereas the implemented method combines aggregate and mode-specific rankings; the final error bound in Eq. (16) depends on the unmeasured support mismatch mim_i. The authors should measure these quantities or provide a more direct analysis of the implemented retrieval procedure.

  5. Corollary 1 requires the re-grounding error to be smaller than the norm of the private target gradient, but Table 1 reports a mean relative error of 1.075 for GRADHOP. Thus, the reported aggregate result does not demonstrate that the condition holds. The target model is also adapted using multiple AdamW steps, while the sufficient condition for final private-objective improvement in Eq. (27) is not evaluated. The authors should report how often these conditions hold or clarify that the theoretical results do not directly explain the observed improvements.

Questions for authors

See Weaknesses.

Reviewer_TvvmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes GradHop, a target-decoupled model adaptation framework that enables clients to contribute private-data-dependent learning information without accessing or optimizing the server's target model. Clients compute aggregate and mode-specific gradient directions using a smaller shared reference model and transmit these directions to the server in a single communication round. The server then retrieves relevant examples from a public repository and uses them to adapt the target model. The paper provides a theoretical analysis of cross-model re-grounding and establishes sufficient conditions under which selected public examples approximate the inaccessible private target gradient. Experiments on Banking77 and CLINC150 demonstrate improvements over public-data-only baselines, with additional evaluations of cross-family transfer and multi-client adaptation.

Critical strengths
  1. Practically relevant problem formulation. The paper addresses the important challenge of adapting large server-side models without requiring resource-constrained clients to access or differentiate through the target model.

  2. Simple and flexible cross-model adaptation mechanism. By using public examples as intermediate learning anchors, GradHop avoids the need for parameter-wise correspondence between reference and target models, making the framework applicable across heterogeneous architectures.

  3. Theoretical motivation and empirical evidence. The paper provides a conditional analysis of gradient approximation and retrieval stability. The experiments demonstrate improved target-gradient alignment and downstream performance in the evaluated intent-classification settings.

Critical weaknesses
  1. Insufficient comparison with stronger retrieval baselines. The main experiments compare GradHop only with random selection, balanced selection, and selection using the full public dataset. Because no comparisons are made with embedding-based nearest-neighbor retrieval or prototype-based selection methods, it remains unclear whether the proposed gradient-based communication mechanism offers substantial advantages over simpler client-conditioned retrieval approaches.

  2. Theoretical guarantees are conditional, and empirical validation is limited. The proposed bounds rely on assumptions concerning the coverage of the public repository, oracle support mismatch, and reference-space separation. Although the experiments demonstrate improved gradient alignment, they do not establish whether these sufficient conditions are satisfied, nor do they indicate how predictive these bounds are in practice.

  3. Insufficient evidence for generalization and scalability. The experiments are largely restricted to intent-classification datasets accompanied by a task-relevant, labeled public repository. The method’s effectiveness remains unclear under significant distribution shifts, limited public-data coverage, and broader generative tasks. In addition, the server-side preprocessing and storage overhead of public gradients are not adequately evaluated.

Questions for authors
  1. When the public support set is matched to the communication budget, how does GradHop compare with simpler client-conditioned retrieval methods—for example, semantic-embedding-similarity-based retrieval, nearest-neighbor selection, or class-prototype-based retrieval?

  2. How sensitive is GradHop to the quality and coverage of the public repository? Specifically, how does the method degrade when the public dataset lacks samples from certain private client classes, or when there is substantial distribution mismatch, or even label-space mismatch?

  3. Can the authors empirically investigate the relationship between the theoretical "re-grounding" bound and practical adaptation performance? In addition, what end-to-end computational, storage, and privacy implications arise from computing and transmitting gradient directions that depend on private data?

Reviewer_yHNROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a target-decoupled adaptation setting: a client holds private data and a small shared reference model, while the server holds a large target model and a PUBLIC repository, and the client never receives, evaluates, or optimizes the target model. The proposed GradHop lets the client form learning directions under the reference model — an aggregate direction plus K mode directions obtained by spherical k-means — and communicates them once. The server retrieves a PUBLIC support by cosine similarity against these directions and adapts the target model directly on the selected examples. The analysis decomposes the target-space error of the selected support into a coverage term and a selection-mismatch term (Proposition 1), derives a tight angular guarantee with respect to the private target gradient (Corollary 1), and gives a multi-step descent condition (Corollary 2), all without assuming parameter-wise correspondence between the two models. Experiments on Banking77 and CLINC150 with Qwen2.5-1.5B as reference and Qwen2.5-7B as target show clear gains over client-independent PUBLIC selection at equal budget, a target-space re-grounding diagnostic (cosine 0.636 vs. 0.258 for the balanced baseline), transfer of the same client message across Mistral/Llama/OLMo target families, and a multiple-client extension. The communicated message is small (~524 KB) and sent once.

Critical strengths
  • A genuinely new and cleanly formulated problem with a complete analysis chain. To tell the truth, I am quite impressed by the decoupling of "where the learning signal is computed" from "where the model is optimized" — a natural but previously unarticulated axis. Each empirical design choice is backed by theory: the error decomposition, the tight angular bound, and the honest one-dimensional counterexample in Appendix B.1 showing that perfect retrieval cannot compensate for a poor PUBLIC repository. This theory-to-method correspondence is exactly how I wish more FL papers were structured, and it is the main reason I lean positive.
  • Convincing mechanism-level evidence. The target-space re-grounding diagnostic directly measures what the theory predicts, and the cross-family transfer experiment — reusing the same client message and PUBLIC supports across Mistral-7B, Llama-3.1-8B, and OLMo-2-7B — is the strongest evidence that the method needs no coordinate-level alignment between reference and target gradient spaces. This substantially de-risks the central claim.
Critical weaknesses
  • Privacy is essentially not analyzed. The client message contains averaged and mode-wise averaged reference gradients of private examples, plus cluster weights that reveal the directional structure of the private data. The gradient inversion literature suggests even aggregated gradients can leak, and for a paper whose motivation is private data, the absence of any empirical inversion attack or honest-but-curious analysis is a material gap.
  • The baseline family is narrow. All empirical competitors are client-independent PUBLIC selection strategies (Random, Balanced, All PUBLIC). Without any comparison to target-side or proxy-based methods operating under less restrictive assumptions (FedKSeed, Ferret, Offsite-Tuning, FedProxy), it is hard to calibrate how much accuracy is sacrificed for target decoupling. Even a single such reference number on the same tasks would substantially change my confidence in the evaluation.
Questions for authors
  • [Clarification] What can an honest-but-curious server infer from the client message, in particular from the mode directions and their weights? Have you run any gradient-inversion or attribute-inference attack on the message?
  • [Clarification] How does GradHop compare, even approximately, to a target-side method (e.g., a zeroth-order or proxy-based approach) on the same tasks, so readers can price the target-decoupling constraint?
  • [Clarification] The PUBLIC repository spans the same 32 labels as the task, so suitable anchors almost surely exist. How does performance degrade under a shifted PUBLIC distribution, and does the degradation track the coverage term as theory predicts?
  • [Clarification] How sensitive is the method to K, B, and Bg, and is there a practical heuristic for setting them?
  • [Clarification] On Llama-3.1-8B, Balanced PUBLIC-128 beats GradHop in accuracy — the only such case. Do you have an explanation?
  • [Suggestion] The All PUBLIC collapse (21.2% on Banking77) suggests indiscriminate adaptation is actively harmful; this finding itself deserves discussion.
  • [Minor] The capitalization of "PUBLIC" throughout reads like a placeholder; the abstract uses "a PUBLIC repository" while the body sometimes uses it as a proper noun — please unify.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.