- Rating
- 2(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates when cross-client collaboration benefits personalized federated learning under heterogeneous data distributions. Focusing on least-squares personalization over fixed shared representations, it characterizes the recipient-specific correction of a population reference direction through two local transfer statistics: a curvature matrix and a feature–objective cross-moment, without requiring sample-level density-ratio estimation. The paper develops local, empirical-Bayes pooled, and structured estimators for these statistics, together with finite-time convergence bounds that account for client–population mismatch and transfer-statistic estimation error. Experiments on synthetic systems, CIFAR-10/100, and FEMNIST features examine the conditions under which collaborative estimation improves personalized learning and where its benefits diminish.
Critical strengths
-
The paper provides an objective-specific characterization of the information required to adapt a population reference direction to individual clients, reducing the mean-correction problem to estimating two finite-dimensional transfer statistics under affine gradients. Its finite-time analysis further connects the estimation accuracy of these statistics to personalized optimization, distinguishing the effects of client–population mismatch, statistical uncertainty, and population-reference error.
-
The paper evaluates the proposed estimators against several personalized federated learning and local-learning baselines across varying heterogeneity and data-scarcity settings. It reports improvements under favorable conditions as well as cases where collaboration fails to outperform strong local estimators, providing a balanced assessment of its empirical performance.
Critical weaknesses
1. Restricted theoretical scope and gaps in coverage of the evaluated methods.
The exact transfer-statistic characterization and main convergence guarantees rely on the affine-gradient setting, limiting their applicability to more general nonlinear personalization. Although Appendix E.4 introduces a historical-gradient estimate and a Gauss–Newton approximation, the resulting approximation error and its effect on optimization are not theoretically bounded. Moreover, the real-feature experiments use a rectangular reference update with a minibatch product bias, while the practical empirical-Bayes pooling uses same-sample weight estimates. Neither implementation is directly covered by Theorem 1, and Proposition 4 analyzes plug-in weights only under an independent-split construction (Appendix F.9).
2. Limited evidence for the distinct benefit of collaborative curvature estimation.
The transfer-statistic characterization exploits the affine-gradient structure, while the pooled and structured estimators build on established statistical estimation techniques. Although the paper provides a finite-time analysis and demonstrates benefits from collaborative estimation in some linear-feature settings, its incremental advantage over simpler correction strategies is less evident beyond the affine-gradient setting. In the nonlinear ablations (Table 11), the client's own running-mean estimator provides substantial improvements, local curvature correction yields smaller additional gains, and federated pooling does not further improve performance in the two reported settings.
Questions for authors
-
How does the minibatch product bias in the rectangular reference update affect convergence, particularly with batch size one? Can the theoretical analysis be extended to the same-sample empirical Bayes pooling weights used in practice?
-
Given the nearly identical performance of local curvature correction and federated pooling in the nonlinear ablations (Table 11), under what conditions does cross-client transfer-statistic estimation provide measurable benefits beyond client-only corrections?
4 comments
Response to Reviewer L38A (2/2)
Q1(a). Minibatch product bias in the rectangular reference update, particularly with batch size one
Response. We thank the reviewer for this precise question. We answer it in Appendix I.5 (p. 37, l. 1955-1978) with Table 21 (p. 38, l. 1998).
Source of the bias. For a minibatch mean of size b, $ E[F̂_bᵀ F̂_b] = F_iᵀ F_i + (1/b) E[(Φ − F_i)ᵀ(Φ − F_i)]$. The second term is a bias. It is largest at b = 1 and vanishes as b grows. Because of it, the same-minibatch update converges to a reference that weights the client objectives by biased Gram matrices (p. 37, l. 1955-1961).
Effect on the correction. The bias affects the reference but not the validity of the correction. Proposition 1 holds for any admissible reference, so the mean field of each client's update remains that of its own objective. This is a statement about the target of the correction. It is not a convergence guarantee for m < d, which Theorem 1 does not cover. When all clients share the sampling distribution, the fixed point of the reference does not move (p. 37, l. 1962-1965).
Measurement. We compared the submitted update with an unbiased cross-product of independent minibatch moments and with a full-moment reference, for b ∈ {1, 8, 32, 128} (p. 37, l. 1966-1969). The submitted update is not harmed by the bias in our experiments (p. 37, l. 1970-1976). With the submitted initialization, which sets the objective reference to the mean of the clients' objective parameters (Appendix H.2, p. 32, l. 1677-1679), the three variants differ by at most 0.2 in the scarce protocol of Table 2, where b = 1. In the with-replacement sweep they agree within one seed standard deviation at every batch size. At b = 1 and δ_env = 0.5, the three values are 33.5 ± 2.2, 33.7 ± 2.5 and 33.1 ± 2.6.
Zero initialization. The one regime where the choice matters is a zero initialization with a single pass. There the unbiased variant is noisier. It is worse under homogeneous sampling by 7-10 and better under skew by 1.4-2.6, while the submitted update and the full-moment reference remain close to each other.
These results are measurements. They do not extend Theorem 1 to m < d (p. 37, l. 1977-1978).
Q1(b). Extension of the analysis to same-sample empirical-Bayes weights
Response. A guarantee for same-sample weights is not yet available, so we measured the gap between the analysed construction and the one we run (Appendix I.5, p. 37, l. 1979-1984, and Table 22, p. 38).
| Setting | Same-sample (submitted) | Cross-fitted | Independent split (Prop. 4) | Oracle |
|---|---|---|---|---|
| CIFAR-10 hom., 128 | 31.6 | 32.1 | 33.7 | 30.6 |
| CIFAR-10 Dir(0.3) | 75.3 | 78.9 | 86.6 | 74.3 |
| CIFAR-100 Dir(0.3) | 54.5 | 57.2 | 60.5 | 53.7 |
Across the eight settings tested, the same-sample weights are within 5% of oracle weights computed from held-out moments, so little remains to be gained. The cross-fitted weights are 0-5% worse than the same-sample weights. The independent-split construction analysed in Proposition 4 costs 5-18%, because each estimator uses only half of the samples. The analysed construction is therefore conservative compared with the one we run.
Q2. Under what conditions does cross-client estimation provide benefits beyond client-only corrections?
Response. We thank the reviewer for this question, which goes to the heart of the paper. Our answer has four parts.
Similar clients. Pooling helps when local samples are scarce relative to the dimension and clients are similar. The gain is optimal up to constants in the estimation term (Proposition 5, p. 35, l. 1872-1882), and it requires sharing the pair of statistics (Appendix I.6, p. 38, l. 2041-2042).
Label skew. Structured sharing helps most when the variability within classes is large compared with the spread between classes, and when the client is not concentrated on classes that are rare in the population. This follows from the exact error formula of Proposition 6 (p. 36-37, l. 1942-1946).
How the statistics are used. With single-pass stochastic iteration, these benefits were not sufficient to improve on local ridge on FEMNIST or under extreme label skew. The one-shot plug-in of the same statistics improves on local ridge in all 20 settings, including those of Table 3 (Section 5.4, p. 9, l. 479-481, and Table 17, p. 33). The collaborative information is therefore present in these settings, and the limitation lies in single-pass stochastic estimation (Section 6, p. 10, l. 537-539).
Please interact with authors
Dear reviewer L38A,
Please respond to the author's rebuttal.
Best,
AC
I appreciate the authors' detailed responses and additional experiments. However, my main concerns remain.
-
Narrow scope and limited practical utility: The theoretical guarantees remain confined to idealized fixed-representation least-squares settings, leaving key practical components (rectangular reference updates, same-sample weights, and nonlinear heads) uncovered. More importantly, the new experiments sharply limit where the proposed iterative framework helps: in nonlinear settings, cross-client pooling provides no benefit and underperforms local Polyak-averaged SGD; even in linear settings, the analyzed iterative methods are consistently outperformed by a simple one-shot closed-form plug-in.
-
Incremental novelty over prior work: As clarified in the revision, the paper directly inherits its problem formulation and control-variate framework from Zhang & Azizan (ICLR 2026), replacing density-ratio correction with moment estimation under affine gradients. Once restricted to fixed-feature least squares—where the strongest empirical variant reduces to closed-form statistical sharing closely aligned with Fed3R—the overall advance appears too incremental for acceptance.
Response to Reviewer L38A (1/2)
We thank the reviewer for a precise and constructive report. The questions on the implementation and on the benefit of collaborative estimation led to several new experiments.
W1. Restricted theoretical scope and gaps in coverage
Response. The reviewer's description is accurate, and the paper lists these points in its scope section (Appendix F.9, p. 29, l. 1534-1548). In the revision we measured both implementation gaps. The rectangular reference update is compared with an unbiased and a full-moment alternative, and the same-sample pooling weights are compared with the analysed independent-split construction and with oracle weights (Appendix I.5, p. 37, l. 1954-1984, with Tables 21 and 22 on p. 38). The results are described under Q1 below. In short, the submitted implementation is not harmed by the product bias in our experiments, and the same-sample weights are within 5% of oracle weights.
For the nonlinear head, the approximation error remains unbounded. The paper presents these experiments as a limited extension of the transfer principle and not as an extension of the main theorem (Appendix E.4, p. 20, l. 1067-1070).
While re-running the experiments, we also noticed that one implementation detail was not documented. The code initializes the objective reference at the mean of the clients' objective parameters. Initializing at zero, or at the mean of the clients' own least-squares estimates, changes the CC results by at most 0.5 in all 20 settings. This is now stated in Appendix H.2 (p. 32, l. 1677-1679).
W2. Evidence for the benefit of collaborative curvature estimation
Response. We thank the reviewer for this observation, which led us to test the contribution of each statistic directly.
Linear heads. We shared the curvature and the cross-moment separately, with everything else fixed (Appendix I.6, p. 37-38, l. 1986-2042, and Table 23, p. 39). The result is summarized in the main text (Section 5.4, p. 10, l. 512-514).
The benefit of collaboration is clear, and it comes from sharing the pair of transfer statistics. For the iterative methods, sharing the cross-moment accounts for nearly all of the improvement, while sharing the curvature alone is slightly worse than sharing nothing in all ten rows of the table (p. 37, l. 1991-1995). For the class-structured plug-in under label skew, each statistic alone is worse than sharing nothing, while sharing both gives 18.4 (p. 38, l. 2032-2037). This is what Proposition 1 predicts, because the correction is determined by the pair and not by either statistic alone (p. 38, l. 2037-2038). The reviewer's point about curvature specifically is therefore well taken, and the appendix states the conclusion in these terms (p. 38, l. 2041-2042).
Nonlinear heads. The reviewer's reading of the component ablation (Table 14 in the revision, p. 21) is correct. For the nonlinear head, the client's own running mean provides the primary estimate of the transfer direction (Appendix E.4, p. 20, l. 1053), and pooling adds no further gain. To check whether this changes with less data, we ran two additional experiments with the same MLP head (Appendix E.4, p. 20, l. 1072-1079). The first re-runs the nonlinear experiment with an additional baseline, Polyak-averaged independent SGD (Table 12, p. 20). The second varies the number of local samples, n_tr ∈ {64, 128, 256, 1024} (Table 13, p. 21).
Pooling does not help at any sample size. The pooled and client-only corrections differ by at most 0.3. The transfer variants improve on plain independent SGD in all ten configurations, and Polyak-averaged independent SGD is the strongest method in all ten. The benefit of collaborative estimation is therefore specific to the linear setting covered by the theory. The main text now says that the nonlinear experiments show no additional benefit from pooling (Section 5, p. 10, l. 517-520).