A Closer Look at Weight and Feature Regularization in Federated Learning

Federated learning trains a shared model through local updates on distributed client data. Many methods address data heterogeneity by encouraging local model weights, features, or predictions to remain close to those of the global model. However, these distances alone do not fully describe how regularization changes the model’s predictions or learning updates. In this paper, we study regularization on weight, feature, and prediction through the forward and backward computations of a neural network. We examine what each penalty directly preserves and how changes in features, weights, and prediction errors propagate to layer outputs, backward signals, and task gradients. This provides a common view of existing formulations and clarifies which changes are constrained and which can remain during local training. Our experiments show that regularizers can affect signals beyond their direct targets, while stronger agreement with the global model does not consistently correspond to higher accuracy. These results are consistent with our analysis, helping explain why a regularizer is useful in some settings but less effective in others.