acceptodds
Withdrawn from ICLR 2027

A Closer Look at Weight and Feature Regularization in Federated Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Federated learning trains a shared model through local updates on distributed client data. Many methods address data heterogeneity by encouraging local model weights, features, or predictions to remain close to those of the global model. However, these distances alone do not fully describe how regularization changes the model’s predictions or learning updates. In this paper, we study regularization on weight, feature, and prediction through the forward and backward computations of a neural network. We examine what each penalty directly preserves and how changes in features, weights, and prediction errors propagate to layer outputs, backward signals, and task gradients. This provides a common view of existing formulations and clarifies which changes are constrained and which can remain during local training. Our experiments show that regularizers can affect signals beyond their direct targets, while stronger agreement with the global model does not consistently correspond to higher accuracy. These results are consistent with our analysis, helping explain why a regularizer is useful in some settings but less effective in others.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Federated learning, Heterogeneous data, Regularization

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_zzgfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Many existing federated learning methods aim to mitigate local drift by regularizing model weights, features, or predictions to remain close to those of the global model. This paper questions the underlying assumption that reducing such discrepancies necessarily leads to effective regularization. Specifically, the authors analyze how different regularization methods influence the forward and backward passes of local training. Their analysis reveals that regularization affects not only the quantities being directly constrained but also various other signals involved in the training process. Furthermore, empirical results demonstrate that reducing the discrepancy from the global model does not necessarily translate into improved performance.

Critical strengths
  1. The paper provides a systematic comparison and analysis of various regularization methods based on their regularization targets, offering a unified perspective on these approaches.

  2. The paper provides an analysis of how regularizing specific model components, including features, weights, and predictions, influences various signals during the forward and backward passes of training.

Critical weaknesses
  1. Lack of clear motivation and intuition. While the paper analyzes the effects of regularization on features, weights, and predictions, the motivation behind these analyses is not sufficiently clear. In particular, it is difficult to understand what specific questions the analyses aim to address or what insights they are intended to provide.
  2. Limited novelty of the conclusions. The main conclusions appear rather obvious. It is already intuitive that regularization affects not only its direct targets but also other signals involved in training, and that simply reducing the discrepancy from the global model does not necessarily lead to better performance.
  3. Limited experimental evaluation. The experiments consider a relatively limited set of baseline methods, and the empirical results do not appear to provide additional insights.
  4. Clarity and presentation. The paper is difficult to follow in several places, making it challenging to identify the main takeaways. In addition, some paragraphs on page 4 appear to repeat similar discussions.
Questions for authors
  1. What is the primary motivation behind the analyses conducted in this paper, and what specific research questions are they intended to address?
  2. What novel insights can be drawn from the theoretical analyses and empirical results presented in this paper?
Reviewer_my9ZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies local-training regularizers in federated learning (FL) that encourage the local model to remain close to the global model at the level of weights, features, or predictions (e.g., FedProx, feature distillation). Rather than proposing a new method, the authors introduce a common formulation for describing such penalties. They then derive, for a general affine layer, decompositions of changes in the forward output, backward signal, and weight gradient into “conditional” contributions, an interaction term, and cross terms, and describe how gradients from feature- and prediction-level penalties propagate to earlier layers. Experiments with ResNet-18 on CIFAR-10/100 and Tiny ImageNet measure these quantities and suggest that penalties affect signals beyond their direct targets, that individual contributions differ from the combined change, and that stronger agreement with the global model does not consistently translate into higher accuracy.

Critical strengths
  1. The underlying idea. Describing local-training regularizers in FL from a unified perspective, by specifying which quantity is modified and which resulting quantity is compared against a reference, is natural and potentially useful.

  2. Careful presentation. The derivations are presented carefully, and the identities and inequalities in Eqs. (15)-(23) and Appendix A appear correct. The numerical claims in the text are consistent with the tables.

  3. Appropriate qualification of claims. The authors consistently acknowledge the limitations of their claims: they distinguish bounds from equalities, clarify that the comparisons in Section 5.1 are not claims about the training process, and state that the coefficients were not tuned.

Critical weaknesses
  1. The proposed unification has no substantive implications.

The unification of regularizers is presented as the central contribution, but in its current form it is a notational convention rather than an analytical tool. The general form presented in the paper covers virtually any penalty that compares a quantity to a reference, and hence makes no claim by itself. Its only concrete consequence, the gradient MΔqM\Delta q, holds only for quadratic distances with linear responses and all other quantities fixed. The KL-based penalties do not admit this form, as the authors acknowledge. MOON, cited in the introduction as a representative feature-level method, is not considered at all.

No substantive results follow from the unification. There is no convergence analysis for the class of regularizers, in the spirit of unified analyses of local methods (e.g., Koloskova et al., ICML 2020; Gorbunov et al., AISTATS 2021). There is also no guidance on when one type of penalty should be preferred over another. The only claim that sounds nontrivial is that penalties on different quantities cannot be made equivalent by tuning coefficients. It reduces to the fact that λΔq\lambda\Delta q is collinear with MΔqM\Delta q only when Δq\Delta q is an eigenvector of MM, an elementary linear-algebra statement for a single step, which the authors admit is not tested experimentally. Similarly, the observation that a penalty on the final feature affects earlier-layer gradients follows directly from backpropagation. Overall, the framework re-describes existing methods but yields no new insight into their behavior or design.

Moreover, most of the facts and conclusions derived in the paper already appear, in one form or another, in prior work, which the paper does not discuss. The matrices E[h0(h0)⊤]\mathbb{E}[h^0(h^0)^\top] and δ0(δ0)⊤\delta^0(\delta^0)^\top arising in weight-output and weight-backward matching are exactly the K-FAC factors of the Fisher matrix (Martens & Grosse, 2015), and the corresponding penalties are essentially the Fisher-weighted proximal terms of EWC (Kirkpatrick et al., 2017) and FedCurv (Shoham et al., 2019). Matching the outputs, Jacobians, or gradients of a reference model rather than its parameters is the idea behind Sobolev training (Czarnecki et al., 2017), Jacobian matching (Srinivas & Fleuret, 2018), and function-space regularization (Benjamin et al., 2019). The decomposition of the prediction KL into true-class and non-target terms is taken from DKD (Zhao et al., 2022), and the resulting interpretation of NTD largely restates the motivation of FedNTD (Lee et al., 2022).

  1. Data heterogeneity is essentially absent from the analysis.

Local regularizers in FL exist precisely to counter client heterogeneity. Without heterogeneity there is no local-update bias, and averaging local models essentially approximates centralized training. Studying these penalties without accounting for heterogeneity is therefore hard to justify.

The theory considers a single client, layer, and input, and compares the local model with the round-start global model. It involves no client distributions, no heterogeneity measure, and no aggregation. All identities in Sections 4-5 hold verbatim for fine-tuning, continual learning, or distillation, and nothing in them is specific to FL.

  1. The experiments illustrate identities rather than test predictions, and the design is not robust.

The theoretical part consists of exact identities and generic bounds, which hold for any model by construction. The experiments therefore cannot confirm or refute them; they can only report the numerical values of the terms involved. For example, Fig. 1 shows that a penalty on the final feature changes the gradients of earlier layers, which is guaranteed by the chain rule. Fig. 2a shows that the cross terms in the gradient decomposition are nonzero, but the decomposition itself holds by construction, and the theory makes no prediction about the sign or magnitude of these terms. The one claim that could be tested, namely that different penalties cannot be made equivalent by tuning their coefficients, is not tested at all. A direct test would compare training with a matrix-weighted penalty (MΔqM\Delta q) against a plain proximity penalty with a tuned scalar coefficient, and check whether the resulting models actually differ.

The central empirical message, that stronger agreement with the global model does not guarantee higher accuracy, is well established. It is visible already in the FedProx paper, where large proximal coefficients degrade performance (Li et al., 2020); it is the main motivation of FedNTD, which argues that full distillation hinders fitting the true class (Lee et al., 2022); and it has been discussed as a trade-off between limiting client drift and allowing local convergence (Mendieta et al., CVPR 2022). In particular, the GKD-versus-NTD comparison in Section 6.4 essentially restates the rationale of FedNTD.

All results are obtained with a single random seed and reported without confidence intervals, although the paper discusses differences of a few tenths of a percentage point. Some reported effects are unusually large: NTD improves over FedAvg by +12.74+12.74 pp on CIFAR-10 Dir(0.1), compared with only +0.20+0.20 pp under IID. Such effects should be confirmed over several seeds before any conclusions are drawn from them.

Questions for authors

See Weaknesses

Reviewer_8tnUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper analyzes how weight, feature, and prediction-level regularizers affect predictions, backward signals, and task gradients in federated learning using ResNet-18 on CIFAR10/100 and Tiny imagenet.

Critical strengths

Paper is well written and the framework setup is clean.

Critical weaknesses

The paper says it isn't proposing a new regularizer. It wants to explain which signals each penalty actually preserves. That's a fair aim, but then the explanation has to be useful in one of two ways: it should predict when a regularizer will help, or it should tell people how to design or tune one. The current version does not answer these two questions.

Additionally, the main finding, "Being closer to the global model does not necessarily imply better performance" is not new. Similar observations have been reported previously, for example, in FedProx-vs-FedAvg type of comparison [FedDyn, SCAFFOLD]. Also for such an analysis to be informative, it should be demonstrated across more diverse experimental settings as the effects of regularization can depend significantly on factors such as heterogeneity levels, the number of devices/clients, architecture, and data types. However, the paper considers only one setup with three standard image datasets.

Overall, my main concern is not the correctness of the analysis, but the significance and generalizability of the work.

Questions for authors

See above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.