acceptodds
Under review as a conference paper at ICLR 2027

Curvature Transfer: When Does Collaboration Help Personalized Learning?

Abstract

Personalized Federated Learning (PFL) improves client-specific models by transferring information across heterogeneous clients, yet existing methods provide limited insight into what recipient-specific information is actually needed for transfer and when collaboration adds value beyond local learning. We characterize this information through local transfer statistics that determine how population information should be adapted to a recipient. For least-squares heads over a shared representation, the characterization is exact and reduces to a curvature matrix and a feature–objective cross-moment, without requiring the full client distribution or client-to-population density ratios. This characterization shows that collaboration is useful when other clients improve estimation of the recipient's transfer statistics through similarity or exploitable structure. We develop pooled and structured estimators for these regimes and provide finite-time guarantees that characterize when collaboration improves personalized transfer. Experiments recover the predicted collaboration regimes and examine whether the transfer principle extends to nonlinear personalized heads. Under scarce homogeneous CIFAR features, pooled estimation reduces error by -% relative to the strongest non-oracle comparator, while structured estimation reduces error by -% under structured CIFAR-10/100 heterogeneity. The results also identify settings in which strong local estimation remains preferable, particularly under higher heterogeneity and on FEMNIST features. Overall, our results characterize when cross-client information improves personalized learning through better estimation of recipient-specific transfer information.

Primary area:
optimization
Keywords:
Personalized Federated Learning, Transferability, Curvature Transfer, Statistical Heterogeneity, Collaborative Estimation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_L38AOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates when cross-client collaboration benefits personalized federated learning under heterogeneous data distributions. Focusing on least-squares personalization over fixed shared representations, it characterizes the recipient-specific correction of a population reference direction through two local transfer statistics: a curvature matrix and a feature–objective cross-moment, without requiring sample-level density-ratio estimation. The paper develops local, empirical-Bayes pooled, and structured estimators for these statistics, together with finite-time convergence bounds that account for client–population mismatch and transfer-statistic estimation error. Experiments on synthetic systems, CIFAR-10/100, and FEMNIST features examine the conditions under which collaborative estimation improves personalized learning and where its benefits diminish.

Critical strengths
  1. The paper provides an objective-specific characterization of the information required to adapt a population reference direction to individual clients, reducing the mean-correction problem to estimating two finite-dimensional transfer statistics under affine gradients. Its finite-time analysis further connects the estimation accuracy of these statistics to personalized optimization, distinguishing the effects of client–population mismatch, statistical uncertainty, and population-reference error.

  2. The paper evaluates the proposed estimators against several personalized federated learning and local-learning baselines across varying heterogeneity and data-scarcity settings. It reports improvements under favorable conditions as well as cases where collaboration fails to outperform strong local estimators, providing a balanced assessment of its empirical performance.

Critical weaknesses

1. Restricted theoretical scope and gaps in coverage of the evaluated methods.

The exact transfer-statistic characterization and main convergence guarantees rely on the affine-gradient setting, limiting their applicability to more general nonlinear personalization. Although Appendix E.4 introduces a historical-gradient estimate and a Gauss–Newton approximation, the resulting approximation error and its effect on optimization are not theoretically bounded. Moreover, the real-feature experiments use a rectangular reference update m<dm<d with a minibatch product bias, while the practical empirical-Bayes pooling uses same-sample weight estimates. Neither implementation is directly covered by Theorem 1, and Proposition 4 analyzes plug-in weights only under an independent-split construction (Appendix F.9).

2. Limited evidence for the distinct benefit of collaborative curvature estimation.

The transfer-statistic characterization exploits the affine-gradient structure, while the pooled and structured estimators build on established statistical estimation techniques. Although the paper provides a finite-time analysis and demonstrates benefits from collaborative estimation in some linear-feature settings, its incremental advantage over simpler correction strategies is less evident beyond the affine-gradient setting. In the nonlinear ablations (Table 11), the client's own running-mean estimator provides substantial improvements, local curvature correction yields smaller additional gains, and federated pooling does not further improve performance in the two reported settings.

Questions for authors
  1. How does the minibatch product bias in the rectangular reference update m<dm<d affect convergence, particularly with batch size one? Can the theoretical analysis be extended to the same-sample empirical Bayes pooling weights used in practice?

  2. Given the nearly identical performance of local curvature correction and federated pooling in the nonlinear ablations (Table 11), under what conditions does cross-client transfer-statistic estimation provide measurable benefits beyond client-only corrections?

4 comments
AuthorsOpenReview ↗

Response to Reviewer L38A (1/2)

We thank the reviewer for a precise and constructive report. The questions on the implementation and on the benefit of collaborative estimation led to several new experiments.

W1. Restricted theoretical scope and gaps in coverage

The nonlinear approximation error is not bounded. […] rectangular reference update […] minibatch product bias […] same-sample EB pooling weights […] neither covered by Theorem 1.

Response. The reviewer's description is accurate, and the paper lists these points in its scope section (Appendix F.9, p. 29, l. 1534-1548). In the revision we measured both implementation gaps. The rectangular reference update is compared with an unbiased and a full-moment alternative, and the same-sample pooling weights are compared with the analysed independent-split construction and with oracle weights (Appendix I.5, p. 37, l. 1954-1984, with Tables 21 and 22 on p. 38). The results are described under Q1 below. In short, the submitted implementation is not harmed by the product bias in our experiments, and the same-sample weights are within 5% of oracle weights.

For the nonlinear head, the approximation error remains unbounded. The paper presents these experiments as a limited extension of the transfer principle and not as an extension of the main theorem (Appendix E.4, p. 20, l. 1067-1070).

While re-running the experiments, we also noticed that one implementation detail was not documented. The code initializes the objective reference at the mean of the clients' objective parameters. Initializing at zero, or at the mean of the clients' own least-squares estimates, changes the CC results by at most 0.5 in all 20 settings. This is now stated in Appendix H.2 (p. 32, l. 1677-1679).

W2. Evidence for the benefit of collaborative curvature estimation

Limited evidence for the distinct benefit of collaborative curvature estimation. […] In the nonlinear ablations (Table 11), the client's own running mean provides most of the correction gain, while federated pooling adds no further benefit.

Response. We thank the reviewer for this observation, which led us to test the contribution of each statistic directly.

Linear heads. We shared the curvature and the cross-moment separately, with everything else fixed (Appendix I.6, p. 37-38, l. 1986-2042, and Table 23, p. 39). The result is summarized in the main text (Section 5.4, p. 10, l. 512-514).

SettingEstimatorNeitherOnly HOnly FBoth
CIFAR-10 hom.CC-PCL (iterative)80.486.033.431.6
CIFAR-10 hom.Plug-in, EB49.049.530.725.8
CIFAR-10 Dir(0.3)CC-class (iterative)80.184.855.653.6
CIFAR-10 Dir(0.3)Plug-in, class53.773.259.418.4

The benefit of collaboration is clear, and it comes from sharing the pair of transfer statistics. For the iterative methods, sharing the cross-moment accounts for nearly all of the improvement, while sharing the curvature alone is slightly worse than sharing nothing in all ten rows of the table (p. 37, l. 1991-1995). For the class-structured plug-in under label skew, each statistic alone is worse than sharing nothing, while sharing both gives 18.4 (p. 38, l. 2032-2037). This is what Proposition 1 predicts, because the correction is determined by the pair and not by either statistic alone (p. 38, l. 2037-2038). The reviewer's point about curvature specifically is therefore well taken, and the appendix states the conclusion in these terms (p. 38, l. 2041-2042).

Nonlinear heads. The reviewer's reading of the component ablation (Table 14 in the revision, p. 21) is correct. For the nonlinear head, the client's own running mean provides the primary estimate of the transfer direction (Appendix E.4, p. 20, l. 1053), and pooling adds no further gain. To check whether this changes with less data, we ran two additional experiments with the same MLP head (Appendix E.4, p. 20, l. 1072-1079). The first re-runs the nonlinear experiment with an additional baseline, Polyak-averaged independent SGD (Table 12, p. 20). The second varies the number of local samples, n_tr ∈ {64, 128, 256, 1024} (Table 13, p. 21).

Test MSE above reference, ×10⁻³Indep.Indep. + PolyakOwn meanCC (local)CC (pooled)
CIFAR-108.0−1.23.83.63.7
FEMNIST34.220.521.320.820.8

Pooling does not help at any sample size. The pooled and client-only corrections differ by at most 0.3. The transfer variants improve on plain independent SGD in all ten configurations, and Polyak-averaged independent SGD is the strongest method in all ten. The benefit of collaborative estimation is therefore specific to the linear setting covered by the theory. The main text now says that the nonlinear experiments show no additional benefit from pooling (Section 5, p. 10, l. 517-520).

AuthorsOpenReview ↗

Response to Reviewer L38A (2/2)

Q1(a). Minibatch product bias in the rectangular reference update, particularly with batch size one

Response. We thank the reviewer for this precise question. We answer it in Appendix I.5 (p. 37, l. 1955-1978) with Table 21 (p. 38, l. 1998).

Source of the bias. For a minibatch mean F^bF̂_b of size b, $ E[F̂_bᵀ F̂_b] = F_iᵀ F_i + (1/b) E[(Φ − F_i)ᵀ(Φ − F_i)]$. The second term is a bias. It is largest at b = 1 and vanishes as b grows. Because of it, the same-minibatch update converges to a reference that weights the client objectives by biased Gram matrices (p. 37, l. 1955-1961).

Effect on the correction. The bias affects the reference but not the validity of the correction. Proposition 1 holds for any admissible reference, so the mean field of each client's update remains that of its own objective. This is a statement about the target of the correction. It is not a convergence guarantee for m < d, which Theorem 1 does not cover. When all clients share the sampling distribution, the fixed point of the reference does not move (p. 37, l. 1962-1965).

Measurement. We compared the submitted update with an unbiased cross-product of independent minibatch moments and with a full-moment reference, for b ∈ {1, 8, 32, 128} (p. 37, l. 1966-1969). The submitted update is not harmed by the bias in our experiments (p. 37, l. 1970-1976). With the submitted initialization, which sets the objective reference to the mean of the clients' objective parameters (Appendix H.2, p. 32, l. 1677-1679), the three variants differ by at most 0.2 in the scarce protocol of Table 2, where b = 1. In the with-replacement sweep they agree within one seed standard deviation at every batch size. At b = 1 and δ_env = 0.5, the three values are 33.5 ± 2.2, 33.7 ± 2.5 and 33.1 ± 2.6.

Zero initialization. The one regime where the choice matters is a zero initialization with a single pass. There the unbiased variant is noisier. It is worse under homogeneous sampling by 7-10 and better under skew by 1.4-2.6, while the submitted update and the full-moment reference remain close to each other.

These results are measurements. They do not extend Theorem 1 to m < d (p. 37, l. 1977-1978).

Q1(b). Extension of the analysis to same-sample empirical-Bayes weights

Response. A guarantee for same-sample weights is not yet available, so we measured the gap between the analysed construction and the one we run (Appendix I.5, p. 37, l. 1979-1984, and Table 22, p. 38).

SettingSame-sample (submitted)Cross-fittedIndependent split (Prop. 4)Oracle
CIFAR-10 hom., 12831.632.133.730.6
CIFAR-10 Dir(0.3)75.378.986.674.3
CIFAR-100 Dir(0.3)54.557.260.553.7

Across the eight settings tested, the same-sample weights are within 5% of oracle weights computed from held-out moments, so little remains to be gained. The cross-fitted weights are 0-5% worse than the same-sample weights. The independent-split construction analysed in Proposition 4 costs 5-18%, because each estimator uses only half of the samples. The analysed construction is therefore conservative compared with the one we run.

Q2. Under what conditions does cross-client estimation provide benefits beyond client-only corrections?

Response. We thank the reviewer for this question, which goes to the heart of the paper. Our answer has four parts.

Similar clients. Pooling helps when local samples are scarce relative to the dimension and clients are similar. The gain is optimal up to constants in the estimation term (Proposition 5, p. 35, l. 1872-1882), and it requires sharing the pair of statistics (Appendix I.6, p. 38, l. 2041-2042).

Label skew. Structured sharing helps most when the variability within classes is large compared with the spread between classes, and when the client is not concentrated on classes that are rare in the population. This follows from the exact error formula of Proposition 6 (p. 36-37, l. 1942-1946).

How the statistics are used. With single-pass stochastic iteration, these benefits were not sufficient to improve on local ridge on FEMNIST or under extreme label skew. The one-shot plug-in of the same statistics improves on local ridge in all 20 settings, including those of Table 3 (Section 5.4, p. 9, l. 479-481, and Table 17, p. 33). The collaborative information is therefore present in these settings, and the limitation lies in single-pass stochastic estimation (Section 6, p. 10, l. 537-539).

Area_Chair_NRiHOpenReview ↗

Please interact with authors

Dear reviewer L38A,

Please respond to the author's rebuttal.

Best,

AC

I appreciate the authors' detailed responses and additional experiments. However, my main concerns remain.

  1. Narrow scope and limited practical utility: The theoretical guarantees remain confined to idealized fixed-representation least-squares settings, leaving key practical components (rectangular reference updates, same-sample weights, and nonlinear heads) uncovered. More importantly, the new experiments sharply limit where the proposed iterative framework helps: in nonlinear settings, cross-client pooling provides no benefit and underperforms local Polyak-averaged SGD; even in linear settings, the analyzed iterative methods are consistently outperformed by a simple one-shot closed-form plug-in.

  2. Incremental novelty over prior work: As clarified in the revision, the paper directly inherits its problem formulation and control-variate framework from Zhang & Azizan (ICLR 2026), replacing density-ratio correction with moment estimation under affine gradients. Once restricted to fixed-feature least squares—where the strongest empirical variant reduces to closed-form statistical sharing closely aligned with Fed3R—the overall advance appears too incremental for acceptance.

Reviewer_sg2LOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates the personalized federated learning problem, and the key setup is a least-squares structure on a shared frozen representation. The insight is that the adaptation requires two component, one is the curvature, and the other is the feature-objective cross-moment. So they estimate them separately locally and propose a corresponding estimation procedure. Non-asymptotic bounds are established, and the experiments on CIFAR and FEMNIST support the claim.

Critical strengths
  1. The idea of replacing the density ratios with two moment matrices and directly estimating from the unlabeled inputs is interesting.

  2. The method shows certain advantages, such as being immune to fresh-sample variance, which is benefited by the proposed structure.

  3. The theoretical results look solid, and the proof looks correct to me. And the paper has a very candid discussion on its scope and its negative regimes, which allows for a fair evaluation.

  4. The experimental results are comprehensive and well support the method.

Critical weaknesses
  1. The heavy dependence of the theoretical results on the least-squares setting limits the broad applicability of the method. All these decoupled components are derived from the least-squares setting. This greatly simplified the theoretical development due to the affine gradient.

  2. The contribution of this paper over the earlier papers like [1] is unclear. It takes the framework and baselines etc from these earlier papers, and it seems like the sole contribution is around things like a better estimate with moment estimates etc. A more detailed discussion and comparison between the proof techniques, theoretical and methodological different shall be added.

  3. The paper, when claiming the collaboration improves personalized learning, is based only on the upper bound. Without a proper lower bound, it is hard to evaluate how much such an improvement is and where is the baseline to compare against.

  4. Some of the results have direct assumptions on rates, which leaves questions about whether they are reachable. Like Proposition 3(c) takes a given estimator with a certain rate to proceed, but does not specify how we obtain such an estimator.

[1] Chenyu Zhang and Navid Azizan. Personalized collaborative learning with affinity-based variance reduction. In International Conference on Learning Representations (ICLR), 2026.

Questions for authors

Please see the weaknesses above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer sg2L (1/2)

We thank the reviewer for the careful assessment of the theory. The requests for a lower bound and for a constructed estimator led to two new propositions.

W1. Dependence on the least-squares setting

The theoretical results rely heavily on the least-squares setting. […] These decoupled components are derived from the special affine gradient.

Response. We agree with the reviewer on this point. The exact characterization and the guarantees rely on the affine gradient of the least-squares head. The paper states this as its scope (abstract, p. 1, l. 16-17, and Section 6, p. 10, l. 531-533), and it presents the nonlinear experiments as a limited extension of the transfer principle rather than an extension of the main theorem (Appendix E.4, p. 20, l. 1067-1070). New experiments confirm that the benefit of collaborative estimation does not carry over to a nonlinear head (Appendix E.4, p. 20, l. 1072-1079). We make no claim of an exact characterization beyond the affine-gradient setting.

Within this setting, the revision strengthens the results in two ways that respond to the reviewer's other points. The estimation rates are shown to be optimal (Proposition 5), and the structured rate is derived for an explicit estimator (Proposition 6). Both are described under W3 and W4 below.

W2. Contribution over Zhang & Azizan (2026)

The contribution of this paper is unclear when compared to Zhang & Azizan (2026). […] A more detailed comparison of the proof techniques should be added.

Response. We thank the reviewer for this request. We have added a dedicated comparison (Appendix I.3, p. 35, l. 1849-1866) with a summary table (Table 20, p. 35).

We inherit four elements from Zhang & Azizan (2026). These are the problem class, the control-variate form of the update, the synthetic benchmark and the projected stochastic-approximation analysis of the reference iterates (p. 35, l. 1851-1854).

Our contribution concerns the information that is needed about the recipient (p. 35, l. 1855-1863). The AffPCL correction evaluates density ratios at other clients' current samples, and their analysis shows that these ratios cannot be estimated at an affinity-adaptive rate without additional structure. Proposition 1 identifies such structure for affine gradients. The required conditional mean depends on the client's distribution only through d2+dmd^2 + dm moments of its own inputs, and this condition is necessary as well as sufficient. Density ratios are therefore one way to build a valid correction, but they are not required.

This changes what clients exchange and how the analysis proceeds. Clients exchange residuals and moments instead of samples for ratio evaluation. On the proof technique, our bounds separate the error of estimating the transfer statistics from the mismatch between a client and the reference (Proposition 2, p. 6). This separation explains why a purely local correction gives no n-dependent gain (Corollary 1, p. 7, l. 366), and it isolates the term that Propositions 3, 5 and 6 control (p. 35, l. 1861-1863).

W3. Lower bound

The claim that collaboration improves personalized learning is based solely on the derived upper bound. Without a lower bound serving as a baseline, it is difficult to effectively evaluate the improvement.

Response. We thank the reviewer for this suggestion. We have added a minimax lower bound for the term through which collaboration acts. The result is Proposition 5 (Appendix I.4, p. 35, l. 1872-1882), with its proof (p. 35-36, l. 1883-1899). It is summarized in the main text (Section 4, p. 6, l. 310-312) and listed among the contributions (Section 1, p. 2, l. 91-92).

Statement. Consider the class of Assumption 1, that is, symmetric AA with mean HH such that λmin⁡(H)≥λx\lambda_{\min}(H) \ge \lambda_x, max⁡{∥H∥, ess sup⁡∥A−H∥}≤GA\max\{\lVert H \rVert,\ \operatorname{ess\,sup} \lVert A - H \rVert\} \le G_A and E∥A−H∥F2≤V\mathbb{E}\lVert A - H \rVert_F^2 \le V. If V≤(GA−λx)/2\sqrt{V} \le (G_A - \lambda_x)/2, then for every N≥1N \ge 1 the minimax risk satisfies

inf⁡H^ sup⁡P EP∥H^−H∥F2 ≥ V24N,\inf_{\hat{H}} \ \sup_{P} \ \mathbb{E}_P \lVert \hat{H} - H \rVert_F^2 \ \ge\ \frac{V}{24N},

where the infimum is over all estimators based on NN samples and the supremum is over the class.

With N=ntN = nt (all clients, shared law) the bound is V/(24nt)V/(24nt). With N=tN = t (one client) it is V/(24t)V/(24t). The proof is a two-point argument.

Consequence. The rates V/tV/t and V/(nt)V/(nt) of Proposition 3(a,b) are optimal up to the constant 24. A purely local estimator of the transfer statistics cannot obtain the nn-fold gain, and pooling cannot be improved beyond a constant (p. 36, l. 1901-1903). This provides the baseline the reviewer asks for, for the estimation term.

The result concerns the estimation of the transfer statistics. It is not a lower bound on the optimization error DtiD_t^i, and it does not address the heterogeneous term. Both points are stated directly after the proposition (p. 36, l. 1903-1904) and in the main text (p. 6, l. 312).

AuthorsOpenReview ↗

Response to Reviewer sg2L (2/2)

W4. Assumed rate in Proposition 3(c)

Some of the theoretical results are assumed rather than formally proven. For example, Proposition 3(c) directly takes an estimator with a given convergence rate […] the paper does not elaborate on how to obtain such an estimator.

Response. We thank the reviewer for this observation. For label skew, we now construct the estimator and compute its error. The setting is described in Appendix I.4 (p. 36, l. 1905-1913), the result is Proposition 6 (p. 36, l. 1914-1923), and its proof follows (p. 36, l. 1925-1940). The result is summarized in the main text (Section 4, p. 6, l. 312-316), with a pointer where the label-skew estimator is introduced (Section 3, p. 5, l. 232).

Estimator. H^i=∑cp^i(c) H^c\hat{H}_i = \sum_c \hat{p}_i(c)\, \hat{H}_c, where the class moments H^c\hat{H}_c are pooled over all clients and p^i\hat{p}_i are the client's empirical class frequencies. This is the estimator used by CC-class (p. 36, l. 1913), so the result applies to a method that we run.

Exact error. Under shared class-conditional laws,

= \mathbb{E}\!\left[\sum_c \hat{p}_i(c)^2\, \frac{V_c}{N_c}\right] + \frac{B_i}{t}, \qquad B_i = \sum_c p_i(c)\, \lVert H_c - H_i \rVert_F^2 .$$ *Rate.* Let $\rho_i = \max_c p_i(c)/\bar{p}(c)$, $V_w = \max_c V_c$, and let $C$ be the number of classes. If $n \ge 2\rho_i$, then $$\mathbb{E}\lVert \hat{H}_i - H_i \rVert_F^2 \le V_w \left[ \frac{4\rho_i}{nt} + \frac{4\rho_i C}{nt^2} + 2\exp\!\left(-\frac{t\, n\, \bar{p}_{\min}}{16}\right) \right] + \frac{B_i}{t}.$$ *Interpretation.* The local estimator has error $\left(\sum_c p_i(c) V_c + B_i\right)/t$. Structured pooling removes the within-class part of this error, at an effective pooled sample size $nt/\rho_i$, and keeps the between-class part $B_i/t$ (p. 36-37, l. 1942-1946). Compared with the form assumed in Proposition 3(c), the shared term carries the factor $\rho_i$, the local term is exactly $B_i/t$, and a minimum class-frequency condition appears (p. 37, l. 1947-1951). A simulation agrees with the identity to within 0.2%. *Scope.* The result covers label skew with shared class-conditional laws. Proposition 3(c) remains a conditional statement for other structures.

Dear Authors,

Thank you for the response. It has addressed my concerns. I will keep my score and support the weak acceptance of the paper.

Reviewer_uoRUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when collaboration improves personalized learning over fixed shared representations. For least-squares heads, it characterizes the recipient-specific correction in a population control-variate update through a curvature matrix and a feature–objective cross-moment. It develops local, pooled, and structured estimators and analyzes how their estimation errors affect personalized optimization. Experiments on synthetic systems and frozen CIFAR/FEMNIST features identify settings where collaboration helps and where strong local estimation remains preferable.

Critical strengths
  1. The paper explicitly identifies the local statistics needed to correct the specified population control variate. Its analysis separates transfer-statistic estimation error from client-to-reference mismatch, clarifying why locally estimating the correction does not automatically produce a client-count-dependent gain.

  2. The experiments include both favorable and unfavorable collaboration regimes, with strong local ridge estimators and an exact-density-ratio oracle as references. The CIFAR improvements and FEMNIST failures provide useful evidence that collaboration requires informative shared structure rather than local sample scarcity alone.

Critical weaknesses
  1. The relationship to existing frozen-encoder statistical-sharing methods is insufficiently discussed. Fed3R [1] aggregates feature second moments and feature–label statistics to solve ridge regression, FedCOF [2] estimates class covariances from client class means, and FedNCM [3] constructs classifiers from aggregated class means. Although these methods primarily target global classifiers, they are closely related to the statistical-sharing premise. The paper should clarify its additional contribution in personalized correction and collaborative estimation beyond this established approach.

  2. The experiments compare against conventional PFL methods but omit these directly relevant statistical-sharing baselines. Local-Ridge uses only local data and therefore does not replace such comparisons. Evaluating compatible variants under the same frozen representations and data budgets, with clearly stated personalization procedures, would help establish whether iterative curvature transfer provides benefits over simpler closed-form alternatives. Accuracy or prediction error should be considered together with communication and computation costs.

  3. Scalability beyond low-dimensional heads remains unclear. The method estimates a d-by-d curvature matrix and a d-by-m cross-moment, whereas the main real-feature experiments use only 64 or 84 dimensions. Larger representations increase both matrix costs and statistical estimation difficulty. The paper acknowledges the storage limitation, but provides limited evidence that the proposed sharing schemes remain efficient and reliable at larger feature dimensions.

  4. The exact characterization applies to an affine-gradient control-variate update over fixed representations, rather than personalized learning in general. Structured estimation additionally requires a suitable reconstruction model, such as shared class-conditional statistics. The nonlinear extension is exploratory, and jointly learned representations remain outside the theory. These boundaries should be reflected more clearly in the interpretation of when collaboration helps.

[1] Accelerating Heterogeneous Federated Learning with Closed-form Classifiers, ICML 2024.

[2] Covariances for Free: Exploiting Mean Distributions for Training-free Federated Learning, NeurIPS 2025.

[3] Guiding The Last Layer in Federated Learning with Pre-Trained Models, 2023.

Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer uoRU (1/4)

We thank the reviewer for the positive assessment of the problem formulation and for the specific suggestions on related work and baselines.

W1. Relationship to frozen-encoder statistical-sharing methods (Fed3R, FedCOF, FedNCM)

The relationship to existing frozen-encoder statistical-sharing methods is insufficiently discussed. […] The paper should clarify its additional contribution in personalized correction and collaborative estimation beyond this established approach.

Response. We thank the reviewer for pointing us to this line of work. We now cite all three papers, position our work against them in the introduction (Section 1, p. 2, l. 62-65), and discuss them in a new related-work paragraph (Appendix B, p. 14, l. 711-724).

The main difference lies in the question being asked. Fed3R, FedNCM and FedCOF aggregate statistics of frozen features across clients in order to fit a single global classifier. We ask what information a particular client needs in order to adapt a population-level direction to its own problem, and under what conditions other clients can help to estimate that information.

Proposition 1 gives the answer for affine gradients. A correction has the required conditional mean if and only if it equals Hixˉ−FiθˉH_i \bar{x} - F_i \bar{\theta}. Two statistics of the recipient are therefore needed. The first is the curvature HiH_i. The second is the feature-objective cross-moment FiF_i. The methods above aggregate feature moments or class means over all clients to build one global classifier. They do not estimate the recipient's pair (Appendix B, p. 14, l. 719-721). Our new experiments confirm that the second statistic matters. Sharing curvature alone improves on local ridge by at most 13% (Section 5.4, p. 9, l. 475-476), and a direct ablation shows that the benefit comes from sharing the pair (Appendix I.6, p. 37-38, l. 1986-2042, and Table 23, p. 39).

The two lines of work are also formally connected. When the objective feature is determined by the class label, as in our benchmarks, the Fed3R ridge classifier combined with the client's own fit θ^i\hat{\theta}_i coincides with the pooled version of our plug-in estimator. We state this in the main text (Section 5.4, p. 9, l. 483-484), in the caption of Table 4 (p. 10, l. 497-499) and in the related-work paragraph (Appendix B, p. 14, l. 721-724). In Table 4, only the class-structured plug-in is labelled as ours. The two approaches differ once the statistics are estimated for the recipient rather than pooled, which is what our class-structured estimator does (p. 9, l. 484-485). Under label skew this difference is large. At Dirichlet(0.3) on CIFAR-10, the pooled plug-in has excess MSE 194.1, compared with 18.4 for the class-structured plug-in (Table 4, p. 10).

Overall, our contribution beyond these methods is threefold: (i) showing that a valid personalized correction is determined by the recipient's pair (Hi,FiH_i, F_i) (Proposition 1), (ii) estimating these statistics for the recipient, through empirical-Bayes pooling and class structure, and (iii) the finite-time analysis. A side-by-side comparison of AffPCL, the three statistical-sharing methods and our work is given in Appendix I.3 (p. 35, l. 1849-1866) and Table 20 (p. 35).

AuthorsOpenReview ↗

Response to Reviewer uoRU (2/4)

W2. Missing baselines, closed-form alternatives and cost

The experimental comparison is incomplete due to the omission of these baselines. […] Compatible variants should be evaluated under the same frozen representations and data budgets […] The comparison should also consider personalization error together with communication and computational costs.

Response. We thank the reviewer for this suggestion, which led to the most informative experiment of the revision. We ran the comparison on all 20 real-feature settings of the submission, with the same features, client partitions, data budgets and seeds. The results are in the new Section 5.4 (p. 9-10, l. 471-516) with Table 4 (p. 10), and the full results are in Table 17 (Appendix I.1, p. 33).

Methods. We evaluate five families of methods, each with a clearly stated personalization procedure (Appendix I.1, p. 32-33, l. 1704-1752). These are (i) Fed3R global ridge on the clients' targets, (ii) curvature-only sharing with the client's own q^i\hat{q}_i, (iii) Fed3R+prox, a proximal local ridge refit around the Fed3R solution, which plays the role of Fed3R with fine-tuning, (iv) the one-shot plug-in of our transfer statistics, defined in Section 3 (p. 5, l. 250-254), and (v) a FedCOF plug-in. FedNCM uses the same class means as FedCOF but returns a nearest-class-mean classifier and no least-squares head. It therefore has no separate entry under our excess-MSE protocol, and we say so in the revision (p. 32, l. 1724-1725).

SettingLocalFed3RCurv.-onlyFed3R+proxCC-PCLCC-classPlug-in, pooledPlug-in, class
CIFAR-10 hom., n = 12851.940.349.433.031.638.624.632.2
CIFAR-10 Dir(0.3)56.1208.567.553.375.353.6194.118.4
FEMNIST hom., 12811.67.814.25.614.916.22.35.1
FEMNIST, 100 writers31.237.636.928.236.037.428.429.7

The comparison gives three results.

First, iterative transfer is better than global Fed3R and than curvature-only sharing. The best CC variant has lower error than both in all 15 CIFAR settings. Global Fed3R breaks down under label skew (208.5 against 56.1 for local ridge), and curvature-only sharing improves on local ridge by at most 13% (Section 5.4, p. 9, l. 475-476). This agrees with Proposition 1, since second moments alone do not transfer the recipient's objective.

Second, Fed3R followed by a proximal local refit is a stronger baseline. Its global solution already combines pooled second moments with pooled feature-label moments, and the refit adapts it to the client. It is better than the best CC variant in 8 CIFAR settings, worse in 6 and tied in one, and it is better on FEMNIST (Section 5.4, p. 9, l. 476-478).

Third, the one-shot plug-in of our transfer statistics is the most accurate method. The better of the pooled and class-structured plug-ins has lower error than the best CC variant in all 20 settings and on every paired seed, by 15-84%. The pooled version suits similar clients and the class-structured version suits label skew. A single fixed variant gives nearly the same result. The class-structured plug-in alone improves on local ridge in all 20 settings and on the best CC variant in 18 (Appendix I.1, p. 33, l. 1760-1761). These include the settings of Table 3, where no iterative method improved on local ridge. The better plug-in is the best deployable method in 19 settings, and on the FEMNIST writer split it is indistinguishable from Fed3R+prox (28.4 ± 2.3 against 28.2 ± 1.4) (Section 5.4, p. 9, l. 479-485).

To answer the reviewer's question directly, on a fixed representation the estimated pair (Hi,Fi)(H_i, F_i) is best used in closed form (p. 9, l. 485). The comparison therefore supports the characterization of Proposition 1. The statistics it identifies give the best estimator we tested, and estimating them for the recipient is what separates the class-structured plug-in from Fed3R under label skew. We have added this finding to the abstract (p. 1, l. 28-30), to the contributions (Section 1, p. 2, l. 95-97) and to the conclusion (Section 6, p. 10, l. 537-539).

AuthorsOpenReview ↗

Response to Reviewer uoRU (3/4)

Communication and computation. As the reviewer asked, we report cost together with error (Appendix I.1, p. 33, l. 1770-1781, and Table 18, p. 34). Communication counts uploads and broadcasts over the whole run. Table 18 reports representative closed-form methods and the three iterative variants. The main rows are reproduced here.

MethodRoundsKiB per client (whole run)Runtime (s)
Fed3R190.5
Fed3R + proximal local ridge191.0
Plug-in, pooled1170.6
Plug-in, class-structured11682.0
Plug-in, FedCOF1868.3
CC-PCL1284111.7
CC-class1283,0682.4

The iterative methods are the most expensive, because they exchange residuals and moments over 128 rounds. The plug-in uses the same transfer statistics but shares them once. Under similar clients, the pooled plug-in uses about 24 times less communication than CC-PCL and is more accurate (24.6 against 31.6). Under label skew, the class-structured plug-in uses about 18 times less communication than CC-class and is the most accurate method by a wide margin (18.4 against 53.6). Sharing the transfer statistics in one shot therefore gives the best trade-off between accuracy and cost in our experiments (p. 33, l. 1778-1779).

The plug-in applies when the representation is fixed and the head is a least-squares head (Section 3, p. 5, l. 251-254). The iterative update remains the form covered by our analysis. In our benchmarks the targets are noise-free functions of the class label, which makes θ^i\hat{\theta}_i easy to estimate, so the margin may be smaller for noisier targets. We state this condition with the results (Appendix I.1, p. 33, l. 1767-1768).

AuthorsOpenReview ↗

Response to Reviewer uoRU (4/4)

W3. Scalability beyond low-dimensional heads

The method's scalability beyond low-dimensional heads remains unclear. […] Experiments with larger heads are needed to validate the method's reliability and efficiency at realistic representation sizes.

Response. We thank the reviewer for raising this point. We extracted 512-dimensional ResNet-18 features of CIFAR-10 and repeated the comparison up to d = 512 with 128 samples per client. The two largest dimensions therefore have fewer local samples than features, which is the hardest regime for local estimation. The results are in Appendix I.2 (p. 34, l. 1816-1835) with Table 19 (p. 34, l. 1795), and they are summarized in the main text (Section 5.4, p. 10, l. 514-516).

d = 64128256512
Hom.: Local-ridge50.873.4116.8178.4
Hom.: CC-PCL31.654.998.6160.7
Hom.: CC-PCL, diagonal curvature31.755.298.9161.0
Hom.: plug-in, pooled24.647.491.7153.6
Dir(0.3): Local-ridge56.277.4107.2133.1
Dir(0.3): CC-class53.674.5103.0130.4
Dir(0.3): plug-in, class18.430.450.972.3
CC-PCL communication (KiB per client)4111,4605,47821,194
Diagonal curvature (KiB per client)96190378754
Plug-in, pooled (KiB per client)17662611,034

Collaboration keeps its benefit as the dimension grows (Appendix I.2, p. 34, l. 1823-1829). Under homogeneous sampling, the absolute gain of CC-PCL over local ridge is stable (19.2, 18.5, 18.2 and 17.7). The relative gain falls from 38% to 10%, because the error of every method grows with d while the sample size is fixed. The plug-in is the most accurate method at every dimension. Under Dirichlet(0.3) skew, the class-structured plug-in reduces the error of local ridge from 133.1 to 72.3 at d = 512, a reduction of 46%.

The communication of CC-PCL grows as d² and reaches about 21 MB per client at d = 512. Two options reduce it (Appendix I.2, p. 34, l. 1830-1835). The plug-in shares the moments once and needs about 1 MB per client at d = 512. A diagonal curvature needs 0.75 MB and matches full pooling under homogeneous sampling at every dimension.

The diagonal variant relies on the PCA basis and on homogeneity. At d = 64 under Dirichlet(0.3) it is worse than full pooling (81.8 against 75.3), as stated in the appendix (p. 34, l. 1833-1835). The O(d²) cost of full curvature estimates is listed as a limitation in the conclusion (Section 6, p. 10, l. 539-540).

W4. Scope of the theory

The theoretical scope is narrower than the paper's broad conceptual framing. […] These boundaries should be more clearly reflected in the interpretation of the theoretical conclusions.

Response. We agree with the reviewer's description of the scope. The exact characterization applies to least-squares heads over a fixed representation, where the affine gradient yields an exact transfer identity. The abstract now states that the representation is fixed (p. 1, l. 17), and the conclusion states the nonlinear finding and identifies jointly learned representations and general nonlinear models as future work (Section 6, p. 10, l. 531-536). The nonlinear experiments are presented as a scope check rather than as validation of the theory (Section 5, p. 10, l. 517-520).

The revision adds three elements that make the boundaries more explicit. First, the new plug-in estimator is introduced with its condition, namely a fixed representation (Section 3, p. 5, l. 251-252). Second, on structured estimation, the reviewer noted that it requires a reconstruction model. Proposition 6 now states this model explicitly for label skew, as shared class-conditional laws, and derives the resulting error (Appendix I.4, p. 36, l. 1905-1923). Third, new nonlinear experiments show that the benefit of collaborative estimation is specific to the linear setting covered by the theory (Appendix E.4, p. 20, l. 1072-1079).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.