acceptodds
Under review as a conference paper at ICLR 2027

Anchor-Guided Personalization for Federated Prompt Learning in Vision-Language Models

Abstract

Federated prompt learning provides a parameter-efficient way to adapt vision-language models while keeping the pre-trained backbone frozen. However, under heterogeneous client distributions, a single global prompt may be too generic to capture client specialization, while sharing personalized prompts keeps knowledge tied to individual clients and requires communicating client-specific representations. This raises a key challenge: how can clients benefit from specialized knowledge learned across the federation while keeping personalized prompts local? To address this challenge, we propose FedAGP (Federated Anchor-Guided Personalization), which enables specialized knowledge sharing through a compact set of group anchors. From these anchors, each client retrieves a class-dependent mixture according to its local characteristics. The resulting retrieval weights are accumulated into anchor usage that further guides server-side aggregation, allowing the same client-anchor relevance to govern both retrieval and refinement without communicating personalized prompts. To improve global coordination, we further introduce consensus-guided global aggregation, which adjusts the contribution of global prompt updates according to their directional agreement. Extensive experiments on 11 heterogeneous federated benchmarks demonstrate that FedAGP achieves a strong balance between personalization and generalization over existing federated prompt learning methods.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Federated Prompt Learning, Vision-Language Models, Personalized Federated Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aw3dOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FedAGP, a federated prompt learning framework for frozen vision-language models that combines a global prompt, shared group anchors, and client-specific prompts. The central objective is to share specialized knowledge across heterogeneous clients while keeping personalized prompts local. Each group anchor is parameterized as a residual around the global prompt. A private signature derived from the client's personalized prompt is used to retrieve a class-dependent mixture of anchor representations. The retrieved representation is combined with the personalized representation, subject to a local class-support mask, and then fused with the global representation through a learned gate. Local visual prototypes provide auxiliary supervision for a client-dominant anchor, while accumulated retrieval weights determine server-side anchor aggregation. The method additionally introduces consensus-guided aggregation of global-prompt updates based on their cosine agreement with the mean update. Experiments cover seven datasets for Local/Base/Novel evaluation, two CIFAR datasets under label heterogeneity, and OfficeHome and DomainNet under feature and label heterogeneity. The principal reported benefit is an improved balance between personalization, cross-client transfer, and novel-class generalization, measured by the three-way harmonic mean of Local, Base, and Novel accuracy.

Critical strengths
  1. A well-motivated mechanism for sharing specialized knowledge across clients.

  2. Substantial reported improvement in the central personalization–generalization trade-off.

  3. A meaningful control separates anchor sharing from additional model capacity.

Critical weaknesses
  1. The central transfer claims lack a complete statistical account, and several reported results have unclear correspondence: the main contribution concerns Local, Base, and Novel accuracy and their three-way harmonic mean. However, Appendix C.1/Table 6 reports three-seed mean and standard deviation only for Local accuracy. It therefore does not establish the variability of Base, Novel, or HM, which are central to the generalization claim. Averaging performance over the last five communication rounds does not characterize variation across independently seeded experiments. Several results also need reconciliation. FedPGP's OxfordPets Local accuracy is 97.29 in Table 1, whereas Table 6 reports 93.77 ± 0.29. Similarly, Table 2 reports 95.38 on CIFAR-10, while Table 11 reports 95.21 for the stated default of M = 6; 95.38 instead appears at M = 2. These differences may reflect different seeds or configurations, but the correspondence is not documented. The authors should identify the configurations and seed sets underlying these tables and report uncertainty for the principal transfer metrics, computing HM within each run before summarizing across runs. This would help assess both the substantial primary HM improvement and the smaller gains in several secondary settings.

  2. The contribution of anchor-specific refinement is not sufficiently isolated from supervision of the shared global prompt. Anchor features are parameterized as hm(c)=g([pg+am,c])h_m^{(c)}=g([p_g+a_m,c]), while Eq. (9) stops gradients only through the visual prototype. Under the joint optimization specified in Eq. (13) and Algorithm 1, the distillation loss therefore produces identical gradients with respect to pgp_g and the selected residual ami∗a_{m_i^*}, for a fixed selected index. Although the remaining residuals receive no direct distillation gradient, their effective anchor prompts also change through the shared pgp_g. This coupling complicates the interpretation that prototype supervision concentrates refinement on the most relevant anchor: the observed gains may arise from selected-residual specialization, additional visual supervision of the global prompt, or their interaction. Tables 4 and 7 demonstrate the usefulness of distillation and dominant-anchor selection, but do not distinguish these effects. The authors should clarify the intended gradient routing and compare the current formulation with residual-only distillation, stopping gradients through pgp_g specifically within the distillation branch while keeping the remaining training procedure fixed. This comparison would more directly substantiate the proposed mechanism of anchor-specific refinement.

  3. Consensus-guided aggregation remains indirectly sensitive to update magnitude, limiting its directional-consensus interpretation: Equation (11) measures cosine agreement with a reference computed from the raw mean of client updates. Although cosine similarity removes magnitude dependence for a fixed reference, the reference itself depends on the magnitudes of all participating updates. For example, consider four updates

\qquad \Delta_4 = -4u,$$ where u is a unit vector. Their mean is -u/4, so the fourth client has agreement score 1 and the other three have agreement score -1. Equation (12) then assigns the fourth client weight $$\gamma_4 = \frac{1}{1 + 3e^{-2/\tau_g}},$$ which is effectively one at the default temperature of 0.05. Thus, a larger update can determine the reference direction and receive almost all the aggregation weight despite opposing the other three clients. This limits the majority-direction interpretation in Figure 2 and the explanation that weighting reflects direction rather than magnitude. Table 8 supports the empirical usefulness of CGA, but does not isolate this dependence. A targeted analysis of update norms and aggregation-weight concentration, or a comparison using a reference constructed from normalized client directions, would clarify when the proposed rule provides the intended coordination.
Questions for authors
  1. Statistical reporting and table correspondence. Can you specify the seeds, configurations, and averaging procedures underlying Tables 1, 2, 6, and 11, and explain the OxfordPets and CIFAR-10 differences identified in W1? Please provide mean and standard deviation for Base, Novel, and HM for FedAGP and the strongest relevant baselines, using matched experimental settings.

  2. Gradient routing in prototype distillation. Does the implementation of Eq. (9) update the global prompt as well as the selected anchor residual? If so, how does the current formulation compare with residual-only distillation when the remaining training procedure is held fixed? If the implementation detaches the global prompt, please make that explicit in the equations and algorithm.

  3. Magnitude dependence of the consensus reference. How unequal are client update norms in the reported settings, and how concentrated are the resulting aggregation weights? Would constructing the reference from normalized update directions preserve the gains over prompt averaging reported in Table 8?

6 comments
AuthorsOpenReview ↗

Reply to Reviewer aw3d

Comment 1/3

We thank the reviewer for the careful reading of our manuscript and for identifying these points. We corrected the table correspondence, clarified the experimental protocol, and added the analyses requested below. Changes are highlighted in blue in the revised PDF, and line references refer to that version.

W1 / Q1: Statistical reporting and table correspondence.

Experimental protocol and seeds. The main results, including Tables 4–5 and Figure 5, use one training run (seed 1) on the default class-to-client assignment and 16-shot samples (Lines 370–374). Tables 7 and 8 use three additional runs with seeds 1, 2, and 3, independently resampling both the class assignment and training samples for each seed (Appendix C.2, Lines 846–851). Table 17 (formerly Table 11) varies MM with N=100N=100 on the default split. Within each run, we average the last five communication rounds. This temporal averaging is distinct from the variation measured across independent seeds.

FedPGP on OxfordPets. Table 1 uses the default partition, whereas Table 7 averages independently resampled class-to-client assignments and 16-shot samples. With roughly two base classes per client on OxfordPets, the particular class assignment can materially affect Local accuracy (Appendix C.2, Lines 849–851).

CIFAR-10 table correction. Table 2 previously contained the M=2M=2 result (95.38). We corrected it to the default M=6M=6 result (95.21 in Line 371), matching Table 17 (Line 1032). FedAGP remains ahead of pFedMoAP (95.01) and pFedMMA (94.92).

Multi-seed Base, Novel, and HM. Appendix C.2 (Lines 846–853 and Table 8, Lines 864–877) now reports all four metrics over three seeds. HM is calculated for each dataset within each run, averaged over datasets, and then summarized across seeds as a mean and standard deviation.

MethodLocalBaseNovelHM
FedPGP87.39±1.3286.47±1.9078.99±1.3983.82±0.90
FedPHA96.63±0.2197.95±0.1454.86±3.0176.00±2.12
pFedMoAP96.46±0.2079.90±0.0577.89±0.5683.48±0.26
FedMGP95.78±0.3395.83±0.0566.78±0.4582.28±0.24
pFedMMA96.82±0.2698.53±0.1175.30±0.2788.13±0.15
FedAGP96.67±0.2598.19±0.2578.76±0.7090.11±0.21

FedAGP achieves 90.11 ± 0.21 HM, compared with 88.13 ± 0.15 for pFedMMA. Its Novel accuracy is 3.46% higher, while Local and Base accuracy remain similar. Table 8 also reports results for Zero-Shot CLIP, PromptFL, and FedOTP.

AuthorsOpenReview ↗

Reply to Reviewer aw3d

Comment 2/3

W2 / Q2: Gradient routing in prototype distillation.

The reviewer is correct that pgp_g is not detached in our distillation branch. Since hmi∗(c)=g([pg+ami∗,c])h_{m_i^{\ast}}^{(c)}=g([p_g+a_{m_i^{\ast}},c]), the distillation loss backpropagates through both pgp_g and the selected residual ami∗a_{m_i^{\ast}}. Other residuals receive no direct distillation gradient, although changes to pgp_g affect their effective prompts. We clarified this in Section 3.2 (Lines 254–256).

We tested the pathways separately by detaching either pgp_g or ami∗a_{m_i^*} from the distillation branch only. The classification loss and all other training settings are unchanged (Appendix C.8, Lines 968–985 and Table 15, Lines 985–993). Values are averaged over the four ablation datasets.

Ldist\mathcal{L}_{\mathrm{dist}} updatesLocalBaseNovelHM
w/o Ldist\mathcal{L}_{\mathrm{dist}}96.3597.9863.6481.31
pgp_g only96.4097.9170.6485.69
ami∗a_{m_i^*} only (residual-only)96.2398.0172.7986.82
pgp_g and ami∗a_{m_i^*} (ours)96.1597.9475.7788.29

Updating only the anchor retains about 79% of the HM gain from distillation (86.82, versus 81.31 without distillation). Updating only pgp_g retains about 63% (85.69). The joint formulation reaches 88.29, above either individual pathway. The anchor pathway has the larger individual contribution, but the two are complementary. We therefore retain joint supervision.

AuthorsOpenReview ↗

Reply to Reviewer aw3d

Comment 3/3

W3 / Q3: Magnitude dependence of the consensus reference.

We agree with the reviewer that the cosine score in Eq. (11) is scale-invariant only when the reference is fixed. Our reference is the raw mean update, so larger updates affect its direction and, in turn, the aggregation weights. We assessed this dependence by replacing the reference with the mean of normalized directions, dˉ t=1∣St∣∑iΔit/∥Δit∥2\bar d^{\,t}=\frac{1}{|\mathcal S_t|}\sum_{i}\Delta_i^t/\|\Delta_i^t\|_2, without changing the final update rule or other settings (Appendix C.5, Lines 911–930).

Update norm variation and weight concentration (Table 10, Lines 931–939)

DatasetNorm ratio (max/min)neffn_{\text{eff}} (of 10)Corr(norm, weight), Eq. (11)Corr(norm, weight), normalized ref.
Caltech1012.1×3.5+0.31+0.01
DTD3.3×2.8+0.44+0.03
Flowers1023.0×4.2−0.08−0.39
UCF1012.3×6.8+0.10−0.32

Update norms differ by a factor of 2.1–3.3 within a round. The effective number of contributing clients, neff=1/∑iγi2n_{\text{eff}}=1/\sum_i\gamma_i^2, is 2.8–6.8 out of 10. Thus, weights are uneven but do not concentrate on a single client in these experiments. With the original reference, norm and weight are positively correlated on three datasets (up to +0.44 on DTD). Normalizing the reference brings the correlation close to zero on Caltech101 and DTD and makes it negative on Flowers102 and UCF101.

Performance with a normalized reference (Table 11, Lines 941–946)

SettingLocalBaseNovelHM
Prompt averaging95.8997.8973.0986.84
CGA, normalized reference95.7797.3673.7186.99
CGA, original reference (Eq. 11, ours)96.1597.9475.7788.29

The normalized-reference variant reaches 86.99 HM, close to prompt averaging (86.84) but below the original CGA (88.29). It also trails the original reference on each of the four datasets. This suggests that magnitude-sensitive reference construction accounts for much of the observed gain over prompt averaging and directional agreement with a normalized reference provides only a small improvement in this comparison.

Manuscript revisions. Section 3.3 (Lines 290–296) now explains that agreement is measured against a collective reference whose direction depends on update magnitudes. We removed the earlier direction-only description and revised the Figure 2 annotation accordingly. Appendix C.5 (Tables 10–11) contains the full comparison.

Reviewer_saQsOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates federated prompt learning for vision-language models (VLMs) under heterogeneous client distributions. To address the dilemma that a single global prompt is often too generic while sharing personalized prompts compromises privacy and restricts cross-client collaboration, the authors propose FedAGP (Federated Anchor-Guided Personalization). FedAGP introduces a compact set of shared group anchors that act as an intermediate knowledge-sharing layer. Clients perform class-dependent retrieval from these anchors using private signatures and refine the most relevant anchors with local visual prototypes. Furthermore, the framework introduces a usage-weighted anchor aggregation and a consensus-guided global aggregation mechanism to mitigate the negative impact of conflicting client updates. Extensive evaluations across 11 heterogeneous federated benchmarks demonstrate that FedAGP achieves a strong balance between local personalization and cross-client generalization.

Critical strengths

Well-motivated intermediate knowledge-sharing architecture: The design of group anchors successfully bridges the gap between the two extremes of "one-size-fits-all" global prompts and isolated personalized prompts. By allowing clients to perform soft, class-dependent retrieval over shared anchors, the method enables flexible knowledge transfer among related clients without directly exposing or exchanging private personalized prompts.

Comprehensive and rigorous empirical validation: The experimental section is exceptionally thorough, covering 11 diverse benchmark datasets that simulate various types of heterogeneity, including base-to-novel generalization, label shift, and domain-shift scenarios. Coupled with extensive ablation studies (e.g., on retrieval weights, feature fusion, and anchor aggregation), the empirical claims are highly convincing and well-supported.

Critical weaknesses

While FedAGP demonstrates strong empirical performance across numerous benchmarks, the paper lacks theoretical analysis regarding its convergence properties under non-IID data distributions. Given the complex interactions between local prompt updates, class-dependent anchor retrieval, and consensus-guided aggregation, providing convergence bounds or analyzing how anchor-guided updates mitigate client drift would significantly strengthen the paper's theoretical foundation.

The consensus-guided global aggregation weights updates based on their directional similarity to the mean update. While this effectively suppresses conflicting updates from majority trends, it runs the risk of disproportionately silencing minority clients or those holding rare, underrepresented classes/domains. A discussion or evaluation on whether this consensus mechanism inadvertently harms the adaptation of rare classes would make the design more robust and convincing.

Questions for authors

Please see weakness

5 comments
AuthorsOpenReview ↗

Reply to Reviewer saQs

Comment 1/3

We thank the reviewer for the time and effort spent reviewing our manuscript and for the thoughtful comments on the theoretical analysis and on minority clients. Our responses are below.

W1: Theoretical analysis under non-IID data.

We agree that the effect of anchor sharing under non-IID data deserves a more explicit analysis. Local objectives may produce conflicting global-prompt updates, while the shared anchors offer a separate channel for client-dependent variation. We added Appendix F (Lines 1123 onward), covering retrieval gradients and local anchor movement, usage-weighted aggregation, and global-prompt disagreement under a quadratic surrogate.

Class-dependent retrieval and local anchor updates. Each client retrieves a class-dependent mixture of shared anchors. For class cc, the retrieved representation is

ri(c)=∑m=1Mαi,m(c)hm(c),r_i^{(c)}=\sum_{m=1}^{M}\alpha_{i,m}^{(c)}h_m^{(c)},

where αi,m(c)\alpha_{i,m}^{(c)} measures the relevance of anchor mm to client ii. Differentiating this representation gives

=\alpha_{i,m}^{(c)}\left[I+\frac{(h_m^{(c)}-r_i^{(c)})\nabla_h\phi^\top}{\tau_a}\right]. \tag{1}$$ Here, $\phi$ is the anchor–signature similarity score. The first term is the anchor's direct contribution and the second accounts for the change in retrieval weights. Both are scaled by $\alpha_{i,m}^{(c)}$, so anchors with greater relevance generally receive stronger retrieval-path gradients. Assuming bounded representations, encoder and fusion derivatives, and local-loss gradients (including prototype supervision), we obtain the following bound after $K$ local steps: $$\sum_{m=1}^{M}\|a_{i,m}^{t,K}-a_m^t\|\leq\eta K G_A, \tag{2}$$ Here, $\eta$ is the learning rate. Under the stated assumptions, $G_A$ does not grow with the number of anchors $M$. The bound describes total local anchor movement rather than convergence of the full method. **Usage-weighted anchor aggregation.** Clients differ in their reliance on individual anchors. Uniform averaging can dilute an anchor's update by combining active users with clients that barely update it. FedAGP uses accumulated retrieval usage to weight the contributions. For anchor $m$, let $\Delta_{i,m}$ denote client $i$'s update, $w_{i,m}=\omega_{i,m}/\sum_j\omega_{j,m}$ its normalized usage weight, $u_i=1/|S_t|$ the uniform weight, and $\bar\Delta_m=\sum_i u_i\Delta_{i,m}$. The aggregate can be written as $$\sum_i w_{i,m}\Delta_{i,m}=s_m\bar\Delta_m+\epsilon_m,\qquad s_m=1+\chi^2(w_m\|u)\geq1, \tag{3}$$ The residual $\epsilon_m=\sum_i(w_{i,m}-s_mu_i)\Delta_{i,m}$ captures differences among client updates. When frequently used anchors receive approximately aligned updates and this residual is small, usage weighting reduces dilution relative to uniform averaging. In Table 4(d), usage-weighted aggregation reaches **88.29%** harmonic mean (HM), compared with **86.41%** for uniform averaging.
AuthorsOpenReview ↗

Reply to Reviewer saQs

Comment 2/3

Shared anchors and global-prompt disagreement. We use a quadratic surrogate to isolate the effect of shared anchors on client disagreement. It assumes fixed retrieval coefficients and common curvature across clients. Let DD denote centered client-preferred prompts, ZZ the centered retrieval patterns, AA the anchor matrix, and HH the common curvature matrix. The resulting global-prompt gradient variance is

\mathcal V(A)=\frac{1}{N}\|(D-ZA)H\|_F^2. \tag{4}

The term ZAZA captures the variation represented by the shared anchors. For optimally fitted anchors A⋆A^\star,

\mathcal V(A^\star)=\mathcal V(0)-\frac{1}{N}\|\Pi_ZDH\|_F^2\leq\mathcal V(0), \tag{5}

where ΠZ\Pi_Z projects onto the column space of ZZ. Under this surrogate, optimally fitted anchors remove the component of client disagreement represented by the retrieval patterns. If clients start from the same global prompt and take one deterministic gradient step, the variance of their updates is η2V(A)\eta^2\mathcal V(A) and Eq. (5) then gives a corresponding reduction in one-step client drift.

Taken together, the analyses clarify distinct roles for retrieval, usage-based aggregation, and shared anchors. Retrieval affects which anchors receive local gradients, usage weights determine whose updates are emphasized, and the surrogate shows how well-fitted anchors can reduce global-prompt disagreement. CGA acts separately on the global-prompt updates.

Scope of the analysis. Eqs. (1)–(3) concern local update and aggregation behavior under explicit assumptions. Eqs. (4)–(5) apply to a simplified quadratic surrogate with fixed retrieval coefficients and common curvature. None of these results establishes convergence of the full, nonlinear FedAGP algorithm. We make this limitation explicit at the start of Appendix F.

AuthorsOpenReview ↗

Reply to Reviewer saQs

Comment 3/3

W2: Effect of CGA on minority clients and rare classes.

We compared FedAGP against a variant that replaces CGA with prompt averaging (FedAGP w/o CGA). Both runs use the same partitions, seed, client sampling, hyperparameters, and test sets. This isolates the effect of CGA on minority-domain and rare-class performance.

Minority domains (DomainNet, 12 clients, 6 domains).

SettingClipartInfographPaintingQuickdrawRealSketchAvg
FedAGP w/o CGA90.1979.8288.6651.4294.6187.8082.08
FedAGP91.0480.8489.6252.7494.6988.3882.88
Δ\Delta (CGA)+0.85+1.02+0.96+1.32+0.08+0.58+0.80

CGA improves accuracy in all six domains, with the largest changes on Quickdraw (+1.32) and Infograph (+1.02).

Rare classes (CIFAR-100, Dir(0.5), 100 clients). Rarity is the number of clients whose training data contain the class.

SettingOverall10 rarest (27–60 clients)25 rarest25 most common (79–86 clients)Worst clientStd over clients
FedAGP w/o CGA80.2582.4281.1378.5869.084.20
FedAGP80.3083.1081.4878.4569.184.56

Accuracy improves by 0.68% for the 10 rarest classes and by 0.35% for the 25 rarest classes, while decreasing by 0.13% for the 25 most common classes. Worst-client accuracy changes little (69.08 to 69.18), although the standard deviation across clients increases from 4.20 to 4.56. The higher absolute accuracy of rare classes reflects this personalized evaluation setting: such classes are present on fewer clients, where they may represent a larger share of the local training data.

Separate channels for specialized knowledge. CGA affects only global-prompt aggregation. Personalized prompts stay local, while anchors are shared through usage-weighted aggregation rather than CGA. Thus, down-weighting a client's global-prompt update does not remove its local prompt or prevent its anchor updates from contributing. In the evaluated settings, we find no evidence of reduced minority-domain or rare-class accuracy due to CGA.


Reviewer_d6mJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes FedAGP, a federated prompt-learning method that combines a global prompt, shared group anchors, and private client prompts. Clients retrieve class-specific mixtures of anchors, while the server aggregates anchors by usage and global updates by directional agreement. Experiments on 11 datasets show a strong balance between personalization and unseen-class generalization.

Critical strengths
  1. Eleven datasets cover local personalization, cross-client transfer, novel-class generalization, label heterogeneity, and domain heterogeneity.
  2. The average harmonic mean across seven datasets improves from 83.90% for pFedMoAP to 90.21%.
  3. Experiments examine retrieval, fusion, aggregation, distillation, anchor count, and label heterogeneity.
  4. The appendix reports trainable parameters, upload/download costs, training time, and GPU memory.
Critical weaknesses
  1. The proposed method excludes the personalized representations for classes absent during inference. It is shown that this improves the generalization in Table 4(c). What remains unclear is how much of the advantage over personalized baselines comes from anchor sharing versus this inference rule. Applying this to compatible baselines would help isolate the contribution.

  2. Performance ablations report the usefulness of anchors, but do not directly establish that they learn distinct, reusable specializations. Visualizing anchor usage across clients, classes, and domains would help assess diversity, collapse, and the relationship between retrieval weights and actual transfer benefits.

  3. Table 12 reports 112 KB upload and 112 KB download, compared with 16 KB each for several methods, which is a substantial increase. Local trainable parameters also increase to 66,560. Although these costs remain small in absolute terms, comparisons at matched communication and parameter budgets would better establish the method’s efficiency–accuracy trade-off.

  4. The comparison omits relevant baselines such as pFedMMA, which also combines shared and personalized adaptation for personalized federated VLMs. Including it under the same evaluation protocol would help establish FedAGP’s advantage over existing personalization methods.

  5. Experiments use CLIP ViT-B/16. Testing another backbone would help establish whether the gains depend on this particular model.

  6. Appendix Table 6 reports three-seed variability for Local accuracy only. However, the main claimed advantage concerns Base/Novel generalization and HM. These metrics, the main ablations, and the smaller gains on domain-shift benchmarks benefit from corresponding uncertainty estimates.

Questions for authors
  1. How does FedAGP compare with pFedMMA under the same evaluation protocol?
  2. Do the gains persist when baselines receive comparable parameter and communication budgets?
  3. How sensitive is FedAGP to client count and participation rate?
5 comments
AuthorsOpenReview ↗

Reply to Reviewer d6mJ

Comment 1/3

We thank the reviewer for the time spent reviewing our manuscript and for the constructive suggestions. We have added targeted ablations and baseline comparisons to address the concerns below. Changes are marked in blue in the revised PDF and all line references refer to that version.

W1: Contribution of anchor sharing and the class-support rule.

Cross-client anchor sharing. We use the Local-only anchors control in Table 4(a) (Lines 432-443, description in Lines 431 and 456–459), now also included in Table 5, to separate anchor sharing from the class-support rule. The variant retains the same anchor capacity, class-wise retrieval, prototype supervision, class-support constraint, and adaptive fusion, but does not aggregate anchors across clients. HM falls from 88.29 to 85.97, and Novel accuracy decreases by 4.58%. Since capacity and the class-support rule are unchanged, the comparison isolates the benefit of cross-client anchor sharing.

Class-support rule on baselines. We applied the same rule to FedPGP and FedOTP at inference, without retraining either method (Table 5, Lines 444–454 and discussion in Lines 474–485). The original checkpoints from Table 1 are used throughout. For observed classes, the original global–personalized representation is retained and for unobserved classes, predictions use only the global prompt. The candidate classes for Local evaluation include unobserved classes. Results are averaged over the four ablation datasets.

MethodSettingLocalBaseNovelHM
FedPGPOriginal87.7684.2172.5580.58
FedPGPSupport-aware75.2582.3275.1476.67
FedOTPOriginal93.4595.8838.1761.09
FedOTPSupport-aware85.0181.8055.2369.83
FedAGPw/o δ\delta95.4097.0470.1084.97
FedAGPw/ δ\delta, local-only anchors96.6697.5371.1985.97
FedAGPw/ δ\delta (ours)96.1597.9475.7788.29

The rule does not improve every method. FedPGP's HM decreases by 3.91%, with Local accuracy falling from 87.76 to 75.25. One likely reason is that FedPGP was trained with the personalized branch active for all classes, so its global-only and personalized scores were not calibrated for this mixed inference setting. FedOTP improves Novel accuracy by 17.06%, but loses Local and Base accuracy, resulting in 69.83 HM. FedAGP instead uses the constraint during training as well as inference: cross-entropy is computed over the full class set, with δ=0\delta=0 for unobserved classes. Even without the constraint, FedAGP reaches 84.97 HM, above both baseline variants.

W2: Anchor specialization and reuse.

Figure 4 (Lines 486–496) shows normalized per-client anchor usage ωi,m\omega_{i,m} under the default setting (M=6M=6, τa=0.05\tau_a=0.05). We added the discussion in Lines 515–521.

On DomainNet, each client uses several anchors. The largest single-anchor share ranges from 0.20 to 0.43, compared with 0.17 for uniform usage, and all six anchors receive nonzero usage. Some clients from the same domain have similar profiles: the two Real clients, for example, have usage vectors (0.17, 0.19, 0.13, 0.20, 0.09, 0.22) and (0.18, 0.16, 0.15, 0.19, 0.09, 0.23). On Caltech101, clients 1, 3, 9, and 10 place most weight on a5a_5 (0.53, 0.42, 0.64, and 0.57), while the others give it relatively little weight (0.04–0.22) and draw more from a2a_2–a4a_4.

These patterns indicate that retrieval does not collapse to one anchor and is not a fixed client- or domain-level assignment. We have also clarified in the manuscript (Lines 515–521) that usage patterns alone cannot establish distinct semantic specializations. The performance comparisons provide complementary evidence for learned retrieval: Table 4(b) reports 88.29 HM for class-wise retrieval, compared with 86.91 for client-wise retrieval, 85.70 for uniform mixing, and 77.20 for a fixed random mapping. The Local-only anchors control further shows the benefit of sharing the learned anchors.

AuthorsOpenReview ↗

Reply to Reviewer d6mJ

Comment 2/3

W3 / Q2: Communication and parameter budgets.

We evaluated FedAGP under the communication budget of PromptFL, FedOTP, FedPGP, and FedPHA (Appendix D, Lines 1043–1053 and Table 18, Lines 1055–1063). The global and personalized prompts and the anchor each use eight context tokens, with one anchor (M=1M=1). Clients upload and download 8×5128\times512 global-prompt and 8×5128\times512 anchor parameters in fp16: 16 KB in each direction per round, slightly less than pFedMMA's 16.5 KB. This setting has 13,312 trainable parameters (three 8×5128\times512 prompts and a 1,024-parameter gate), compared with 16,384 for FedOTP and FedPHA. Other hyperparameters remain unchanged.

MethodTrainable paramsUpload / Download (KB)Avg LocalAvg HM
pFedMoAP74,24016 / 16096.6883.90
FedMGP12,80025 / 1094.2782.20
pFedMMA674,56016.5 / 16.596.9288.10
FedAGP (default)66,560112 / 11296.6090.21
FedAGP (16 KB)13,31216 / 1696.6589.54

At the matched communication budget of 16 KB, FedAGP achieves an HM of 89.54%, only 0.67% below its default configuration, while outperforming all baselines. This includes pFedMMA (88.10% HM), which trains approximately 50 times more parameters, and methods with the same communication budget (at most 83.54% HM). Although this configuration does not evaluate multi-anchor retrieval, it demonstrates that FedAGP remains effective under substantially reduced communication and parameter budgets.

W4 / Q1: pFedMMA comparison.

We included pFedMMA (ICLR'26), re-implemented from the official repository with the same evaluation protocol and ViT-B/16 backbone (Lines 356–361 and implementation and cost details in Appendix E, Lines 1102–1112).

BenchmarkMetricpFedMMAFedAGP
7 base-to-novel datasets (avg.)Local96.9296.60
Base98.0597.75
Novel75.4979.91
HM88.1090.21
CIFAR-10, Dir(0.5)Acc.94.9295.21
CIFAR-100, Dir(0.5)Acc.79.7980.30
OfficeHome (avg.)Acc.88.7790.05
DomainNet (avg.)Acc.82.6782.88

Among the baselines, pFedMMA has the highest average HM and slightly higher Local and Base accuracy than FedAGP. FedAGP improves Novel accuracy by 4.42% and HM by 2.11%, and leads on five of the seven datasets while pFedMMA leads on OxfordPets and DTD. Full results are provided in Table 1 (Lines 324–336), Table 2 (Lines 362–371), and Table 3 (Lines 378–385), with discussion in Lines 397–399.

AuthorsOpenReview ↗

Reply to Reviewer d6mJ

Comment 3/3

W5: Backbone evaluation.

We repeated the seven base-to-novel experiments with frozen CLIP ViT-B/32, using the ViT-B/16 hyperparameters without retuning (Appendix C.3, Lines 857–881 and Table 9, Lines 883–897). The seven-dataset averages are:

MethodLocalBaseNovelHM
Zero-Shot CLIP68.8573.3377.6873.03
PromptFL76.2179.7875.3776.91
FedOTP84.8086.2616.8234.87
FedPGP83.7482.2475.5079.97
FedPHA95.4997.1351.1271.50
pFedMoAP95.1384.9876.0884.30
FedMGP94.1895.9154.2073.55
pFedMMA96.0897.6970.1884.64
FedAGP95.7097.3074.6887.15

FedAGP achieves 87.15 HM, 2.51% above pFedMMA, and leads on five of the seven datasets. It maintains high Local and Base accuracy while keeping Novel accuracy (74.68) relatively close to zero-shot CLIP (77.68). The same balance observed with ViT-B/16 therefore holds with ViT-B/32.

W6: Uncertainty estimates for Base, Novel, and HM.

The main tables use one run (training seed 1) with the default class-to-client assignment and 16-shot samples (Lines 370–374). Appendix C.2 reports Local, Base, Novel, and HM over three additional runs with seeds 1, 2, and 3. Each seed determines both training randomness and a separately resampled data split (Lines 846–853 and Table 8, Lines 864–877).

MethodLocalBaseNovelHM
Zero-Shot CLIP72.69±0.3176.07±0.5681.30±0.0276.42±0.09
PromptFL79.69±0.5082.66±1.1879.00±0.2880.27±0.23
FedOTP92.78±0.3794.76±0.1637.71±1.2860.55±1.86
FedPGP87.39±1.3286.47±1.9078.99±1.3983.82±0.90
FedPHA96.63±0.2197.95±0.1454.86±3.0176.00±2.12
pFedMoAP96.46±0.2079.90±0.0577.89±0.5683.48±0.26
FedMGP95.78±0.3395.83±0.0566.78±0.4582.28±0.24
pFedMMA96.82±0.2698.53±0.1175.30±0.2788.13±0.15
FedAGP96.67±0.2598.19±0.2578.76±0.7090.11±0.21

FedAGP records the highest HM (90.11 ± 0.21), compared with 88.13 ± 0.15 for pFedMMA. This 1.98% HM advantage is accompanied by a 3.46% improvement in Novel accuracy, while Local and Base accuracy are similar. The method ordering is broadly consistent with Table 1, aside from a few closely matched baselines.

Q3: Sensitivity to client count and participation rate.

We varied the number of clients and participation rate separately on CIFAR-100 with Dir(0.5)\mathrm{Dir}(0.5), leaving all hyperparameters fixed. An asterisk marks the default Table 2 setting (N=100N=100, 10% participation).

(a) Number of clients, with 10% participation per round

MethodN=50N=50 (5 clients/round)N=100N=100* (10 clients/round)
FedPHA77.7675.90
pFedMoAP80.7879.45
pFedMMA80.9679.79
FedAGP81.1580.30

(b) Participation rate, with N=100N=100 clients

Method5% (5 clients/round)10%* (10 clients/round)20% (20 clients/round)
FedPHA74.8875.9076.65
pFedMoAP78.9679.4579.78
pFedMMA79.0579.7980.11
FedAGP79.3680.3080.37

FedAGP leads in every setting, by 0.19–0.51% over pFedMMA. Accuracy generally increases when there are fewer clients, which leaves more data per client, or when a larger share of clients participates in each round. The relative ordering of methods remains stable. Table 17 (Lines 1026–1033) also shows that a small anchor bank works with N=100N=100: at M=2M=2, FedAGP reaches 95.38% on CIFAR-10 and 80.16% on CIFAR-100. These results do not suggest a need to scale the anchor bank linearly with client count.


Comments on the paper

1 comment
AuthorsOpenReview ↗

Reply to all Reviewers

Dear Reviewers,

Thank you for the careful reviews and constructive suggestions. We appreciate that you found the shared-anchor design a well-motivated way to share specialized knowledge between a single global prompt and isolated personalized prompts (saQs, aw3d), recognized the improved balance between personalization and generalization (d6mJ, aw3d), and considered the evaluation comprehensive, with 11 datasets and extensive ablations (d6mJ, saQs). We have addressed these points in the individual responses below.

We uploaded a revised manuscript, with all changes highlighted in blue. The main revisions are:

  • pFedMMA baseline (ICLR'26) (Tables 1–3, Lines 324–385). FedAGP achieves 90.21 HM across the seven base-to-novel datasets, compared with 88.10 for pFedMMA, and achieves higher accuracy on CIFAR-10/100, OfficeHome, and DomainNet.
  • Class-support constraint and anchor sharing (Table 5, Lines 444–454 and 474–485). Applying the rule to FedPGP and FedOTP does not reproduce FedAGP's performance. Retaining the rule but disabling cross-client anchor sharing reduces HM from 88.29 to 85.97.
  • Anchor usage (Figure 4, Lines 486–496 and 515–521). Clients draw on multiple anchors, with usage patterns that vary across clients.
  • Multi-seed evaluation (Appendix C.2, Lines 846–853; Table 8, Lines 864–877; protocol in Lines 370–374). We report Local, Base, Novel, and HM across three independently resampled data splits.
  • ViT-B/32 evaluation (Appendix C.3, Lines 857–881; Table 9, Lines 883–897). Without hyperparameter retuning, FedAGP reaches 87.15 HM, compared with 84.64 for the strongest baseline.
  • CGA magnitude sensitivity (Appendix C.5, Tables 10–11, Lines 911–946). We analyze the effect of update magnitudes and clarify the description in Section 3.3 (Lines 290–296) and Figure 2.
  • Distillation gradient routing (Appendix C.8, Table 15, Lines 968–993). We identify the gradient pathways explicitly in Section 3.2 (Lines 254–256) and evaluate each pathway separately.
  • Matched communication budget (Appendix D, Table 18, Lines 1043–1063). A 16 KB configuration with 13,312 trainable parameters reaches 89.54 HM, above the compared baselines.
  • Theoretical analysis (Appendix F, Lines 1123 onward). We examine local anchor updates, usage-weighted aggregation, and global-prompt disagreement under stated assumptions.
  • Table correction (Line 371). Table 2 now reports the default M=6M=6 CIFAR-10 result (95.21), consistent with Table 17 (Line 1032).

We sincerely appreciate your time and feedback, which have helped us improve the paper substantially. We hope the revisions address your concerns and would be glad to clarify any remaining questions during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.