pFedAER: Adaptive Expert Retrieval for Personalized Federated Prompt Learning

Personalized federated prompt learning (pFedPL) exploits clients' lightweight prompts as experts to exchange knowledge for performance improvement while keeping the local vision-language model (VLM) frozen. In general, the effectiveness of pFedPL hinges on how well the server routes which remote prompts are worth communicating to each client. Existing approaches rely on fixed-size retrieval of nearest-neighbor prompts based on a symmetric and time-invariant metric of prompt proximity. However, performance gain in reality is directed, evolves over the course of training, and could be negative. To bridge this gap, we formulate prompt retrieval as a problem of communication-constrained online transfer and propose an adaptive expert retrieval method (pFedAER) for personalized prompt mixture. Specifically, on the client side, each participant measures the signed loss improvement of the gated mixture over its local expert and distributes this credit across selected experts via class-conditional gate attention. On the server side, this feedback is used to estimate directed client-expert utility, to balance exploitation against uncertainty and freshness, and to adapt the number of downloaded experts under a hard communication budget. Furthermore, a persistent negative-transfer guard filters client-expert pairs that are repeatedly harmful. Evaluated on nine datasets and two backbones, pFedAER outperforms competing baselines in several settings, especially under combined feature and label shift. In addition, in a controlled study with observable transfer utilities, pFedAER reduces mean regret from $0.144$ to $0.032$ relative to Euclidean retrieval, lowers the rate of harmful selections from $39.2\%$ to $8.7\%$, and consumes only $74.2\%$ of the communication budget. Our implementation code will be available publicly.