What Does the Global Model Leave Behind? Client-Conditioned Residual Experts for Federated Learning

Federated learning typically treats a trained global model as the endpoint of collaborative optimization, yet even strong shared models can leave structured, client-dependent residual errors. We ask what predictive structure remains after shared learning and whether it can be corrected without relearning the deployed model. We introduce FedCoREx (Federated Client-Conditioned Residual Experts), which freezes a deployment-selected shared model and federatively learns a bank of shared residual experts together with a client-conditioned router. The router combines frozen sample representations, base prediction probabilities, and train-only client contexts summarizing the base model's class-conditional prediction behavior to produce additive logit corrections, while preserving the original predictor as an always-on substrate. We further characterize the residual correction problem theoretically: the unrestricted log-loss headroom of a frozen predictor decomposes into sample-only model misspecification and additional client-conditioned information, while the gain realized by FedCoREx also depends on the expressiveness of its residual family and the effectiveness of learning. This perspective explains why statistical heterogeneity alone does not imply correctable residual structure. Across controlled and natural federated settings, FedCoREx yields substantial improvements when strong client-conditioned residual structure is exposed, while producing modest or near-neutral corrections in settings where little usable residual headroom is captured by the current context and correction family. Mechanism controls and evaluations across multiple shared-model substrates further support the roles of conditional context and residual composition. These results position residual correction as a complementary layer to global federated optimization rather than a replacement for it.