Trust Without Aggregation: Robust Expert Pools for Personalized Federated Prompt Learning

Personalized federated prompt learning can preserve client heterogeneity by retaining uploaded prompts as separately retrievable experts instead of averaging them into a single model. However, this design worsens robustness: an expert pool delivers each prompt intact to its recipients, so a malicious prompt that remains geometrically close to benign ones can pass selection while encoding targeted behavior. We formulate this setting as robust expert-pool learning and propose pFedREP, a staged trust-governance mechanism that matches each defense to the evidence available at its stage, i.e., neighborhood-relative screening at cold start, recipient-directed counterfactual utility and distinct-reporter reputation as interaction evidence accumulates, diversity-aware retrieval, and prompt-independent capped routing that bounds reliance on any selected expert. Under fixed retrieval and ordering, we prove that replacing one normalized non-local expert shifts each class logit by at most a cap-controlled constant, yielding a margin condition for top-1 stability. Empirically, across multiple benchmarks, the trust governance of pFedREP does not sacrifice clean accuracy under heterogeneous settings. More importantly, under the primary attack, clean and worst-decile accuracy change by only 0.11 and 0.08 percentage points. In addition, a 20-client study isolates the individual trust mechanisms: relative to geometric screening alone, the full method reduces targeted exposure from 65.3% to 1.3%; and relative to nearest-prompt retrieval, it reduces severe negative transfer from 17.3% to 0.6%. These results support staged trust and bounded reliance as complementary principles for governing unaggregated expert pools. The code will be open-sourced.