Anchor-Guided Personalization for Federated Prompt Learning in Vision-Language Models

Federated prompt learning provides a parameter-efficient way to adapt vision-language models while keeping the pre-trained backbone frozen. However, under heterogeneous client distributions, a single global prompt may be too generic to capture client specialization, while sharing personalized prompts keeps knowledge tied to individual clients and requires communicating client-specific representations. This raises a key challenge: how can clients benefit from specialized knowledge learned across the federation while keeping personalized prompts local? To address this challenge, we propose FedAGP (Federated Anchor-Guided Personalization), which enables specialized knowledge sharing through a compact set of group anchors. From these anchors, each client retrieves a class-dependent mixture according to its local characteristics. The resulting retrieval weights are accumulated into anchor usage that further guides server-side aggregation, allowing the same client-anchor relevance to govern both retrieval and refinement without communicating personalized prompts. To improve global coordination, we further introduce consensus-guided global aggregation, which adjusts the contribution of global prompt updates according to their directional agreement. Extensive experiments on 11 heterogeneous federated benchmarks demonstrate that FedAGP achieves a strong balance between personalization and generalization over existing federated prompt learning methods.