- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes FedAGP, a federated prompt learning framework for frozen vision-language models that combines a global prompt, shared group anchors, and client-specific prompts. The central objective is to share specialized knowledge across heterogeneous clients while keeping personalized prompts local. Each group anchor is parameterized as a residual around the global prompt. A private signature derived from the client's personalized prompt is used to retrieve a class-dependent mixture of anchor representations. The retrieved representation is combined with the personalized representation, subject to a local class-support mask, and then fused with the global representation through a learned gate. Local visual prototypes provide auxiliary supervision for a client-dominant anchor, while accumulated retrieval weights determine server-side anchor aggregation. The method additionally introduces consensus-guided aggregation of global-prompt updates based on their cosine agreement with the mean update. Experiments cover seven datasets for Local/Base/Novel evaluation, two CIFAR datasets under label heterogeneity, and OfficeHome and DomainNet under feature and label heterogeneity. The principal reported benefit is an improved balance between personalization, cross-client transfer, and novel-class generalization, measured by the three-way harmonic mean of Local, Base, and Novel accuracy.
Critical strengths
-
A well-motivated mechanism for sharing specialized knowledge across clients.
-
Substantial reported improvement in the central personalization–generalization trade-off.
-
A meaningful control separates anchor sharing from additional model capacity.
Critical weaknesses
-
The central transfer claims lack a complete statistical account, and several reported results have unclear correspondence: the main contribution concerns Local, Base, and Novel accuracy and their three-way harmonic mean. However, Appendix C.1/Table 6 reports three-seed mean and standard deviation only for Local accuracy. It therefore does not establish the variability of Base, Novel, or HM, which are central to the generalization claim. Averaging performance over the last five communication rounds does not characterize variation across independently seeded experiments. Several results also need reconciliation. FedPGP's OxfordPets Local accuracy is 97.29 in Table 1, whereas Table 6 reports 93.77 ± 0.29. Similarly, Table 2 reports 95.38 on CIFAR-10, while Table 11 reports 95.21 for the stated default of M = 6; 95.38 instead appears at M = 2. These differences may reflect different seeds or configurations, but the correspondence is not documented. The authors should identify the configurations and seed sets underlying these tables and report uncertainty for the principal transfer metrics, computing HM within each run before summarizing across runs. This would help assess both the substantial primary HM improvement and the smaller gains in several secondary settings.
-
The contribution of anchor-specific refinement is not sufficiently isolated from supervision of the shared global prompt. Anchor features are parameterized as , while Eq. (9) stops gradients only through the visual prototype. Under the joint optimization specified in Eq. (13) and Algorithm 1, the distillation loss therefore produces identical gradients with respect to and the selected residual , for a fixed selected index. Although the remaining residuals receive no direct distillation gradient, their effective anchor prompts also change through the shared . This coupling complicates the interpretation that prototype supervision concentrates refinement on the most relevant anchor: the observed gains may arise from selected-residual specialization, additional visual supervision of the global prompt, or their interaction. Tables 4 and 7 demonstrate the usefulness of distillation and dominant-anchor selection, but do not distinguish these effects. The authors should clarify the intended gradient routing and compare the current formulation with residual-only distillation, stopping gradients through specifically within the distillation branch while keeping the remaining training procedure fixed. This comparison would more directly substantiate the proposed mechanism of anchor-specific refinement.
-
Consensus-guided aggregation remains indirectly sensitive to update magnitude, limiting its directional-consensus interpretation: Equation (11) measures cosine agreement with a reference computed from the raw mean of client updates. Although cosine similarity removes magnitude dependence for a fixed reference, the reference itself depends on the magnitudes of all participating updates. For example, consider four updates
Questions for authors
-
Statistical reporting and table correspondence. Can you specify the seeds, configurations, and averaging procedures underlying Tables 1, 2, 6, and 11, and explain the OxfordPets and CIFAR-10 differences identified in W1? Please provide mean and standard deviation for Base, Novel, and HM for FedAGP and the strongest relevant baselines, using matched experimental settings.
-
Gradient routing in prototype distillation. Does the implementation of Eq. (9) update the global prompt as well as the selected anchor residual? If so, how does the current formulation compare with residual-only distillation when the remaining training procedure is held fixed? If the implementation detaches the global prompt, please make that explicit in the equations and algorithm.
-
Magnitude dependence of the consensus reference. How unequal are client update norms in the reported settings, and how concentrated are the resulting aggregation weights? Would constructing the reference from normalized update directions preserve the gains over prompt averaging reported in Table 8?
6 comments
Reply to Reviewer aw3d
Comment 2/3
W2 / Q2: Gradient routing in prototype distillation.
The reviewer is correct that is not detached in our distillation branch. Since , the distillation loss backpropagates through both and the selected residual . Other residuals receive no direct distillation gradient, although changes to affect their effective prompts. We clarified this in Section 3.2 (Lines 254–256).
We tested the pathways separately by detaching either or from the distillation branch only. The classification loss and all other training settings are unchanged (Appendix C.8, Lines 968–985 and Table 15, Lines 985–993). Values are averaged over the four ablation datasets.
| updates | Local | Base | Novel | HM |
|---|---|---|---|---|
| w/o | 96.35 | 97.98 | 63.64 | 81.31 |
| only | 96.40 | 97.91 | 70.64 | 85.69 |
| only (residual-only) | 96.23 | 98.01 | 72.79 | 86.82 |
| and (ours) | 96.15 | 97.94 | 75.77 | 88.29 |
Updating only the anchor retains about 79% of the HM gain from distillation (86.82, versus 81.31 without distillation). Updating only retains about 63% (85.69). The joint formulation reaches 88.29, above either individual pathway. The anchor pathway has the larger individual contribution, but the two are complementary. We therefore retain joint supervision.
Reply to Reviewer aw3d
Comment 3/3
W3 / Q3: Magnitude dependence of the consensus reference.
We agree with the reviewer that the cosine score in Eq. (11) is scale-invariant only when the reference is fixed. Our reference is the raw mean update, so larger updates affect its direction and, in turn, the aggregation weights. We assessed this dependence by replacing the reference with the mean of normalized directions, , without changing the final update rule or other settings (Appendix C.5, Lines 911–930).
Update norm variation and weight concentration (Table 10, Lines 931–939)
| Dataset | Norm ratio (max/min) | (of 10) | Corr(norm, weight), Eq. (11) | Corr(norm, weight), normalized ref. |
|---|---|---|---|---|
| Caltech101 | 2.1× | 3.5 | +0.31 | +0.01 |
| DTD | 3.3× | 2.8 | +0.44 | +0.03 |
| Flowers102 | 3.0× | 4.2 | −0.08 | −0.39 |
| UCF101 | 2.3× | 6.8 | +0.10 | −0.32 |
Update norms differ by a factor of 2.1–3.3 within a round. The effective number of contributing clients, , is 2.8–6.8 out of 10. Thus, weights are uneven but do not concentrate on a single client in these experiments. With the original reference, norm and weight are positively correlated on three datasets (up to +0.44 on DTD). Normalizing the reference brings the correlation close to zero on Caltech101 and DTD and makes it negative on Flowers102 and UCF101.
Performance with a normalized reference (Table 11, Lines 941–946)
| Setting | Local | Base | Novel | HM |
|---|---|---|---|---|
| Prompt averaging | 95.89 | 97.89 | 73.09 | 86.84 |
| CGA, normalized reference | 95.77 | 97.36 | 73.71 | 86.99 |
| CGA, original reference (Eq. 11, ours) | 96.15 | 97.94 | 75.77 | 88.29 |
The normalized-reference variant reaches 86.99 HM, close to prompt averaging (86.84) but below the original CGA (88.29). It also trails the original reference on each of the four datasets. This suggests that magnitude-sensitive reference construction accounts for much of the observed gain over prompt averaging and directional agreement with a normalized reference provides only a small improvement in this comparison.
Manuscript revisions. Section 3.3 (Lines 290–296) now explains that agreement is measured against a collective reference whose direction depends on update magnitudes. We removed the earlier direction-only description and revised the Figure 2 annotation accordingly. Appendix C.5 (Tables 10–11) contains the full comparison.
Reply to Reviewer aw3d
Comment 1/3
We thank the reviewer for the careful reading of our manuscript and for identifying these points. We corrected the table correspondence, clarified the experimental protocol, and added the analyses requested below. Changes are highlighted in blue in the revised PDF, and line references refer to that version.
W1 / Q1: Statistical reporting and table correspondence.
Experimental protocol and seeds. The main results, including Tables 4–5 and Figure 5, use one training run (seed 1) on the default class-to-client assignment and 16-shot samples (Lines 370–374). Tables 7 and 8 use three additional runs with seeds 1, 2, and 3, independently resampling both the class assignment and training samples for each seed (Appendix C.2, Lines 846–851). Table 17 (formerly Table 11) varies M with N=100 on the default split. Within each run, we average the last five communication rounds. This temporal averaging is distinct from the variation measured across independent seeds.
FedPGP on OxfordPets. Table 1 uses the default partition, whereas Table 7 averages independently resampled class-to-client assignments and 16-shot samples. With roughly two base classes per client on OxfordPets, the particular class assignment can materially affect Local accuracy (Appendix C.2, Lines 849–851).
CIFAR-10 table correction. Table 2 previously contained the M=2 result (95.38). We corrected it to the default M=6 result (95.21 in Line 371), matching Table 17 (Line 1032). FedAGP remains ahead of pFedMoAP (95.01) and pFedMMA (94.92).
Multi-seed Base, Novel, and HM. Appendix C.2 (Lines 846–853 and Table 8, Lines 864–877) now reports all four metrics over three seeds. HM is calculated for each dataset within each run, averaged over datasets, and then summarized across seeds as a mean and standard deviation.
FedAGP achieves 90.11 ± 0.21 HM, compared with 88.13 ± 0.15 for pFedMMA. Its Novel accuracy is 3.46% higher, while Local and Base accuracy remain similar. Table 8 also reports results for Zero-Shot CLIP, PromptFL, and FedOTP.