- The proposed unification has no substantive implications.
The unification of regularizers is presented as the central contribution, but in its current form it is a notational convention rather than an analytical tool. The general form presented in the paper covers virtually any penalty that compares a quantity to a reference, and hence makes no claim by itself. Its only concrete consequence, the gradient MΔq, holds only for quadratic distances with linear responses and all other quantities fixed. The KL-based penalties do not admit this form, as the authors acknowledge. MOON, cited in the introduction as a representative feature-level method, is not considered at all.
No substantive results follow from the unification. There is no convergence analysis for the class of regularizers, in the spirit of unified analyses of local methods (e.g., Koloskova et al., ICML 2020; Gorbunov et al., AISTATS 2021). There is also no guidance on when one type of penalty should be preferred over another. The only claim that sounds nontrivial is that penalties on different quantities cannot be made equivalent by tuning coefficients. It reduces to the fact that λΔq is collinear with MΔq only when Δq is an eigenvector of M, an elementary linear-algebra statement for a single step, which the authors admit is not tested experimentally. Similarly, the observation that a penalty on the final feature affects earlier-layer gradients follows directly from backpropagation. Overall, the framework re-describes existing methods but yields no new insight into their behavior or design.
Moreover, most of the facts and conclusions derived in the paper already appear, in one form or another, in prior work, which the paper does not discuss. The matrices E[h0(h0)⊤] and δ0(δ0)⊤ arising in weight-output and weight-backward matching are exactly the K-FAC factors of the Fisher matrix (Martens & Grosse, 2015), and the corresponding penalties are essentially the Fisher-weighted proximal terms of EWC (Kirkpatrick et al., 2017) and FedCurv (Shoham et al., 2019). Matching the outputs, Jacobians, or gradients of a reference model rather than its parameters is the idea behind Sobolev training (Czarnecki et al., 2017), Jacobian matching (Srinivas & Fleuret, 2018), and function-space regularization (Benjamin et al., 2019). The decomposition of the prediction KL into true-class and non-target terms is taken from DKD (Zhao et al., 2022), and the resulting interpretation of NTD largely restates the motivation of FedNTD (Lee et al., 2022).
- Data heterogeneity is essentially absent from the analysis.
Local regularizers in FL exist precisely to counter client heterogeneity. Without heterogeneity there is no local-update bias, and averaging local models essentially approximates centralized training. Studying these penalties without accounting for heterogeneity is therefore hard to justify.
The theory considers a single client, layer, and input, and compares the local model with the round-start global model. It involves no client distributions, no heterogeneity measure, and no aggregation. All identities in Sections 4-5 hold verbatim for fine-tuning, continual learning, or distillation, and nothing in them is specific to FL.
- The experiments illustrate identities rather than test predictions, and the design is not robust.
The theoretical part consists of exact identities and generic bounds, which hold for any model by construction. The experiments therefore cannot confirm or refute them; they can only report the numerical values of the terms involved. For example, Fig. 1 shows that a penalty on the final feature changes the gradients of earlier layers, which is guaranteed by the chain rule. Fig. 2a shows that the cross terms in the gradient decomposition are nonzero, but the decomposition itself holds by construction, and the theory makes no prediction about the sign or magnitude of these terms. The one claim that could be tested, namely that different penalties cannot be made equivalent by tuning their coefficients, is not tested at all. A direct test would compare training with a matrix-weighted penalty (MΔq) against a plain proximity penalty with a tuned scalar coefficient, and check whether the resulting models actually differ.
The central empirical message, that stronger agreement with the global model does not guarantee higher accuracy, is well established. It is visible already in the FedProx paper, where large proximal coefficients degrade performance (Li et al., 2020); it is the main motivation of FedNTD, which argues that full distillation hinders fitting the true class (Lee et al., 2022); and it has been discussed as a trade-off between limiting client drift and allowing local convergence (Mendieta et al., CVPR 2022). In particular, the GKD-versus-NTD comparison in Section 6.4 essentially restates the rationale of FedNTD.
All results are obtained with a single random seed and reported without confidence intervals, although the paper discusses differences of a few tenths of a percentage point. Some reported effects are unusually large: NTD improves over FedAvg by +12.74 pp on CIFAR-10 Dir(0.1), compared with only +0.20 pp under IID. Such effects should be confirmed over several seeds before any conclusions are drawn from them.