acceptodds
Under review as a conference paper at ICLR 2027

Understanding Head Geometry and Dynamics in Federated Regression through a Natural Solution Selection Rule: An Unconstrained Feature Model Analysis

Abstract

In federated averaging, local objectives can admit multiple optimal heads, making the aggregate depend on which heads clients return. We study this ambiguity in federated multivariate regression with private backbones and a shared linear head, using an unconstrained feature model (UFM) that treats training-sample features as free variables. We introduce a natural selection rule: each client returns the optimal head closest to the broadcast head. We show that global minimization with a vanishing proximal penalty on the head realizes this rule. When the clients' optimal Gram matrices and the initial shared Gram matrix are positive definite, the shared Gram matrix follows a closed recursion and converges to the unique Bures–Wasserstein barycenter of the clients' optimal Gram matrices. Even with this alignment, the limit generally differs from the centralized optimal Gram matrix. We decompose this gap into three positive-semidefinite terms arising from differences in client target means, covariance heterogeneity, and averaging the aligned heads. A correction based on a one-time exchange of target means and covariances recovers the centralized optimal Gram matrix in one round under exact local optimization and the same selection rule. We verify these results numerically in the UFM and test its predictions on five tabular and five image regression datasets using deep networks with feature regularization and long local training. In these experiments, ordinary training approaches the predicted barycenter, while a weak proximal penalty improves endpoint agreement and yields trajectories that closely follow the predicted Gram dynamics. The correction moves the final Gram matrices close to the centralized UFM prediction.

Primary area:
learning theory
Keywords:
unconstrained feature model, neural collapse, federated learning, multivariate regression

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_7m7YOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies ambiguity in federated multivariate regression with private backbones and a shared linear head, using an unconstrained feature model (UFM) that treats training-sample features as free variables. The authors propose a natural solution selection rule and show that the shared Gram matrix converges to the Bures–Wasserstein barycenter of clients’ optimal Gram matrices. This paper further analyzehe the gap to centralized training and its one-round correction from shared target moments.

Critical strengths

1.This paper derives a closed recursion for the shared Gram matrix and proves that it converges to the unique Bures–Wasserstein barycenter of the clients’ optimal Gram matrices under the positive definite assumptions.​ 2.The authors verify their results numerically in the UFM and test predictions on five tabular and five image regression datasets using deep networks with feature regularization and long local training.

Critical weaknesses

1.The main results assume that clients reach their local optima in every round. The assumption differ substantially from practical federated learning, where local training is incomplete, participation is intermittent. 2.The correction requires a one-time exchange of the clients’ target means and covariances. However, the paper does not sufficiently establish the communication cost, scalability, or privacy implications of sharing these statistics in practical deployments.

Questions for authors

1.The analysis relies on positive definite optimal Gram matrices. Can the convergence and selection results be extended to positive-semidefinite or rank-deficient Gram matrices? 2.What are the communication and computational costs of exchanging target means and covariances, particularly when the output dimension is large or the number of clients increases? Could the authors further discuss practical strategies to mitigate these coosts and evaluate the scalability of the proposed correction under such settings? 3.The authots assume that clients reach their local optima in every round , which may not always hold in practical federated learning systems. Could the authors discuss how violations of this assumption may affect the proposed method and whether the main conclusions can be expected to hold under more realistic training settings? 4.The correction relies on the target means and covariance matrices estimated from local data. In practice, these estimates may be unreliable when some clients have only a limited number of training samples. How would such estimation errors affect the effectiveness of the proposed correction, particularly under data-imbalanced settings?

9 comments
AuthorsOpenReview ↗

Reply to Reviewer 7m7Y (1/5): Weakness 1, first part, and Q3 (incomplete local training)

We thank Reviewer 7m7Y for the very constructive comments. We answer them one by one below.

Reviewers LsbS (W, Q1, S1) and 6dL2 (W) raised a similar point.

Answer. Within the UFM framework of the paper, we extended the analysis from local optima in every round to any number EE of local gradient steps per round, and trained DNNs with EE from 1 to 4000 local epochs on ten datasets. To the two parts of Q3:

  • Do the main conclusions hold? With many local epochs, yes: the Gram matrix of the shared head ends near the barycenter G⋆G_\star, also without the proximal term. One reason: with bias and features at their optima, small enough gradient steps never move a head along its fiber (the heads with the same Gram matrix), so training returns the closest optimal head of Theorem 1 up to a second-order error near the client optimum (Appendix H.5 of the revised PDF). With fewer, it ends between G⋆G_\star and GcenG_{\rm cen}, the optimal Gram matrix of centralized training, and the UFM predicts its position.
  • How is the proposed method affected? The correction is affected through the bias. With a trained bias and short local training it misses GcenG_{\rm cen}; with the bias fixed at the global target mean and the correction adapted to it, the centralized solution is kept at every local budget.

Theory (Appendix G.5). Every client takes EE gradient steps of size η\eta per round on the head and the bias; client optima are positive definite, as in the submission.

  • Main conclusions (Theorem 3, Proposition 3). A round of one step is a gradient step on the averaged objective, so the centralized solution is fixed. For a small local budget ηE\eta E the rounds have a unique, locally attracting fixed point Gcen+η(E−1)ΔGG_{\rm cen}+\eta(E-1)\Delta_G, up to second order in ηE\eta E, with ΔG\Delta_G in closed form from the target moments; training the features by gradient steps as well gives the same shift. For large budgets the rounds approach the exact local optimization of Theorem 1, whose limit is G⋆G_\star; in between, the fixed points are computed numerically.
  • The correction (Proposition 4). The paper's correction CmC_m is built for clients that fit their own bias, as at a local optimum. In short rounds a trained bias stays near the global target mean μg\mu_g, and rounds of one step then fix a different matrix GoverG_{\rm over}, the optimum when the scatter of the client target means is counted twice. With the bias fixed at μg\mu_g and the correction Cm′C_m' adapted to it (the same moments, taken about μg\mu_g), the centralized state minimizes every corrected local objective, so local steps of any number leave it fixed.

DNNs. Local SGD with EE epochs per round and 32,000 local epochs in total; medians over three seeds (Supplementary Text S.12 in the supplementary material zip file). The table gives, for ordinary training, the direction error dd (the distance between Gram matrices scaled to unit norm) of the final Gram matrix, as ranges over the ten datasets.

EEdd to GcenG_{\rm cen}dd to G⋆G_\star
17.73×10−47.73\times10^{-4}–4.63×10−34.63\times10^{-3}2.59×10−12.59\times10^{-1}–6.46×10−16.46\times10^{-1}
51.28×10−31.28\times10^{-3}–1.32×10−21.32\times10^{-2}2.53×10−12.53\times10^{-1}–6.42×10−16.42\times10^{-1}
504.31×10−24.31\times10^{-2}–1.11×10−11.11\times10^{-1}2.01×10−12.01\times10^{-1}–5.99×10−15.99\times10^{-1}
4002.45×10−12.45\times10^{-1}–5.66×10−15.66\times10^{-1}5.77×10−35.77\times10^{-3}–3.90×10−13.90\times10^{-1}
40002.65×10−12.65\times10^{-1}–6.71×10−16.71\times10^{-1}3.02×10−33.02\times10^{-3}–3.21×10−23.21\times10^{-2}
  • Main conclusions. At E≤5E\le5 all ten datasets end closest to GcenG_{\rm cen}, and at E=4000E=4000 closest to G⋆G_\star. Over the 110 dataset–budget cells of Figure 5, the trace coordinate of the DNNs (0 at GcenG_{\rm cen}, 1 at G⋆G_\star) differs from its UFM value by 3.78×10−33.78\times10^{-3} in median and by at most 3.81×10−23.81\times10^{-2}. The only fitted number is the ratio r=10r=10 of the feature to the head learning rate, set once on the tabular runs with E≥50E\ge50.
  • The correction CmC_m with a trained bias. At E=1E=1 it ends within 5.92×10−45.92\times10^{-4} of GoverG_{\rm over} on Walker, HalfCheetah, Hopper and Beijing (corrected runs carry the proximal weight ρ=10−3\rho=10^{-3}).
  • The bias fixed at μg\mu_g with Cm′C_m' (same four datasets, eleven budgets). The direction error to GcenG_{\rm cen} is at most 8.65×10−48.65\times10^{-4}. The own-data test error (each test sample predicted by its own client) stays within 5% of that of ordinary training at E=5E=5 (-3.20% to +4.68% over the 44 dataset–budget cells), whereas under ordinary training it is 1.55–7.02 times as high at E=4000E=4000. Fixing the bias alone, without Cm′C_m' or a proximal term, removes most of that excess and leaves the error at E=4000E=4000 5.36–49.5% above the E=5E=5 level.

Scope. Private backbones with a shared head; the theory is for deterministic updates, the DNNs use local SGD. The image and Swimmer runs at E=4000E=4000 were still moving toward G⋆G_\star.

AuthorsOpenReview ↗

Reply to Reviewer 7m7Y (2/5): Weakness 1, second part (intermittent participation)

Reviewer LsbS (S1) raised a similar point.

Answer. With the correction, the theory shows that intermittent participation does not move the Gram matrix: it is the centralized GcenG_{\rm cen} after every round, for any pattern of participating clients, including clients that are absent for several rounds. Without the correction, random participation keeps the Gram matrix fluctuating; for equally weighted clients it stays around the barycenter G⋆G_\star, at a distance set by two factors: one decreases with the number of participants and is 0 when all MM clients take part, and the other grows with the heterogeneity of the clients. Within the UFM framework of the paper, we extended the results to rounds in which only some clients participate, and trained DNNs with several numbers of participants per round; the DNNs follow both results.

Theory (Appendix H.3 of the revised PDF; proofs in Supplementary Text S.8 in the supplementary material zip file). The theory assumes exact local optimization and closest-head selection, as in Theorem 1. It treats the rounds with the correction first, and then the rounds without it.

  • Proposition 8 (with the correction). Absent clients do not affect the corrected rounds. The reason is that the paper's correction CmC_m uses the moments of all MM clients, exchanged once, so clients that have not yet participated are included. All participants of a round then minimize the same objective from the same broadcast and return the unique optimal head closest to this broadcast, so Gt=GcenG_t=G_{\rm cen} for every t≥1t\ge1 and for any nonempty sets of participants. The shared bias follows the participants' target means; fixing it at the global target mean μg\mu_g, with the adapted correction Cm′C_m' of reply 1/5, keeps head and bias both fixed.
  • Theorem 6 and Proposition 9 (without the correction). Each set of participants now has its own barycenter, and a round moves the Gram matrix toward the barycenter of its participants. If k<Mk<M clients are drawn independently and uniformly in each round and the client Gram matrices differ, this target keeps changing, so GtG_t has almost surely no limit. The size of its motion around G⋆G_\star can still be computed. For equal weights and clients with close Gram matrices, the long-run mean-square BW distance to G⋆G_\star is

M−kk(M−1)TrMA\frac{M-k}{k(M-1)}\mathrm{Tr}\mathcal M_A

up to higher-order terms: a sampling factor that is 0 at full participation, times the head scatter TrMA=∑mpmdBW2(G⋆,Gm)\mathrm{Tr}\mathcal M_A=\sum_mp_md_{\rm BW}^2(G_\star,G_m) of the clients' optimal Gram matrices GmG_m.

DNNs. The DNNs agree with both parts of the theory. With the correction and half of the clients per round, the final Gram matrix is within 2.06×10−42.06\times10^{-4} of GcenG_{\rm cen} (relative Frobenius error) in all 21 runs on seven datasets. Without it, the seed-median mean-square BW distance to G⋆G_\star over rounds 9–16 is 0.760 to 1.21 times the formula evaluated on each run's moments, and 0.672 to 1.22 times for 16 further combinations of dataset and kk; a factor of 1 would be exact agreement. Protocol: AdamW, E=4000E=4000 local epochs per round, 16 rounds, proximal weight ρ=10−3\rho=10^{-3}, three seeds (Supplementary Text S.12).

In short, the correction removes the effect of intermittent participation on the Gram matrix, exactly in the UFM and to within the error above in the DNNs. Without the correction, this effect has the size given by the formula above.

AuthorsOpenReview ↗

Reply to Reviewer 7m7Y (3/5): Q1 (positive-semidefinite and rank-deficient Gram matrices)

Reviewers LsbS (S2) and 6dL2 (Q2) raised a similar point.

Answer. Yes, in the theory both results extend to arbitrary positive-semidefinite Gram matrices (Appendix H.1 of the revised PDF, within the UFM framework of the paper). A client is called fully active when its optimal Gram matrix GmG_m is positive definite, and rank deficient when GmG_m is singular: its targets then vary too little in some direction (variance at most λHλW\lambda_H\lambda_W), and the regularization switches this direction off. Selection: the closest optimal head is still unique at every round, from a full-row-rank initial head. Convergence: the shared Gram matrix still converges to a minimizer of the same BW objective. Full activity made this minimizer unique; with rank-deficient clients it is unique outside special configurations. DNNs with rank-deficient clients end at the predicted limit on four of five datasets, and on all five with the correction.

The selection result (Lemma 3 in Appendix H.1). The client optimum is Gm=(αΣm1/2−λHI)+G_m=(\alpha\Sigma_m^{1/2}-\lambda_HI)_+ with α=λH/λW\alpha=\sqrt{\lambda_H/\lambda_W}, where (⋅)+(\cdot)_+ sets negative eigenvalues to zero: the formula of the submission with the switched-off directions set to zero. The optimal head closest to a broadcast head WW is unique if and only if WW⊤WW^\top is positive definite on the directions that the client keeps, and it then has a closed form, as in Lemma 1. With an initial head of full row rank, this holds at every round.

The convergence result (Theorem 4 in Appendix H.1). For arbitrary positive-semidefinite GmG_m and an initial head of full row rank, every round decreases the BW objective V(X)=∑mpmdBW2(X,Gm)\mathcal V(X)=\sum_mp_md_{\rm BW}^2(X,G_m) of Theorem 1, and the shared Gram matrix converges to a minimizer of V\mathcal V. This limit can be singular.

What changes: uniqueness of the limit (Theorem 5 in Appendix H.1). With positive-definite GmG_m, V\mathcal V has a single minimizer, the barycenter G⋆G_\star. With rank-deficient GmG_m it can have several, and the limit can then depend on the initial head. Call the range of GmG_m the support of client mm: the directions it keeps. The minimizer is unique (i) if at least one client is fully active (the submission assumed all); (ii) for almost every configuration of the supports, whatever the eigenvalues and the weights, even if every GmG_m is rank deficient; (iii) if three supports are pairwise distinct hyperplanes. Example: two clients that each keep one direction, orthogonal to each other; V\mathcal V then fixes the diagonal of the limit but not its off-diagonal entry, which the initial head determines.

DNNs. In the main experiments (λH=λW=0.01\lambda_H=\lambda_W=0.01) every client is fully active, so we raised both penalties to λ=0.2\lambda=0.2 and 0.350.35 on the five image datasets; the second column counts the rank-deficient clients. The other columns give the direction error dd of the final Gram matrix to the predicted limit: the minimizer of V\mathcal V without the correction; with it, the optimum shared by all corrected clients, which one round still returns (Proposition 5 in Appendix H.1). The minimizer is unique in every instance, by Theorem 5(i) where a client is fully active and by (iii) where none is. Local SGD, E=4000E=4000, eight rounds, proximal weight ρ=0.1λ\rho=0.1\lambda; medians over three seeds (Supplementary Text S.12 in the supplementary material zip file).

datasetrank-deficient clients at λ=0.2\lambda=0.2, 0.350.35no correction, λ=0.2\lambda=0.2no correction, λ=0.35\lambda=0.35with correction, λ=0.2\lambda=0.2with correction, λ=0.35\lambda=0.35
MPIIGaze6 of 8, 8 of 83.58×10−43.58\times10^{-4}1.23×10−41.23\times10^{-4}5.05×10−55.05\times10^{-5}7.09×10−57.09\times10^{-5}
300W-LP6 of 8, 7 of 88.98×10−48.98\times10^{-4}6.53×10−46.53\times10^{-4}4.84×10−54.84\times10^{-5}8.58×10−58.58\times10^{-5}
dSprites6 of 8, 8 of 88.04×10−48.04\times10^{-4}8.47×10−48.47\times10^{-4}3.44×10−53.44\times10^{-5}4.17×10−54.17\times10^{-5}
3D Shapes6 of 8, 8 of 87.15×10−47.15\times10^{-4}1.96×10−31.96\times10^{-3}4.73×10−54.73\times10^{-5}2.61×10−52.61\times10^{-5}
CIFAR-geo0 of 4, 4 of 44.68×10−34.68\times10^{-3}1.25×10−11.25\times10^{-1}–6.63×10−56.63\times10^{-5}
  • Without the correction. On the four eight-client datasets the final Gram matrix is within 1.97×10−31.97\times10^{-3} of the minimizer at both penalties. Where the minimizer is singular (λ=0.35\lambda=0.35 on MPIIGaze, dSprites and 3D Shapes), the final Gram matrices are singular as well (smallest over largest eigenvalue at most 2.90×10−62.90\times10^{-6}). CIFAR-geo ends farther away because of the finite proximal weight: with the same weight the UFM gives 5.17×10−35.17\times10^{-3} and 1.27×10−11.27\times10^{-1} (Supplementary Text S.6).
  • With the correction. All ten corrected instances (the table and CIFAR-geo at λ=0.30\lambda=0.30) end within 8.58×10−58.58\times10^{-5} of the shared optimum.

Scope. The theorems are for exact local optimization with closest-head selection.

Reviewer_6dL2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how averaging locally optimal heads affects federated regression when clients have private backbones. It shows that selecting the optimal head closest to the global head leads to convergence toward a Bures–Wasserstein barycenter, which generally differs from the centralized solution. The authors characterize this gap and propose a moment-based correction to recover the centralized optimum in one round.

Critical strengths
  1. The paper identifies an important ambiguity in federated averaging: locally optimal heads can differ even when their Gram matrices are identical.

  2. The decomposition of the gap to centralized training into three positive-semidefinite terms is insightful. It explains where the discrepancy comes from and motivates a concrete moment-based correction.

Critical weaknesses

The main convergence result relies on exact local optimization and an explicitly imposed proximal selection rule. But standard FedAvg does not necessarily satisfy it. Therefore, it's unclear whether the derived BW dynamics characterize actual federated training or mainly the proposed idealized procedure.

Questions for authors
  1. Under what conditions would ordinary gradient-based local training implicitly select the optimal head closest to the broadcast head? Can the BW convergence result hold without explicitly imposing the proximal penalty?

  2. How essential is the fully active regime? If some clients have rank-deficient optimal Gram matrices, does the closest-head selection still induce a unique limiting BW barycenter, or can the convergence behavior change fundamentally?

  3. Since the correction makes the clients' profiled objectives identical to the centralized objective, how much of the one-round recovery result follows directly from this equivalence? What additional insight does the proposed correction provide beyond matching the local objectives?

  4. Even when the corrected shared head satisfies Gt+1=GcenG_{t+1}=G_{\mathrm{cen}}, the private backbones may still produce different representations. What's the relationship between Gram matrix recovery and the gap in actual predictions or training objectives relative to centralized learning?

6 comments
AuthorsOpenReview ↗

Reply to Reviewer 6dL2 (1/4): Weakness, part 1 (do the BW dynamics describe actual federated training?)

We thank Reviewer 6dL2 for the very constructive comments. We answer them one by one below.

Reviewers LsbS (W, Q1, S1) and 7m7Y (W1, Q3) raised a similar point.

Answer. Within the UFM framework of the paper, we extended the analysis from exact local optimization to any number EE of local gradient steps without a proximal term, and trained DNNs on ten datasets, with EE from 1 to 4000. The BW dynamics describe the long-local-training end of actual training: at E=4000E=4000 the Gram matrix of the shared head ends near the BW barycenter G⋆G_\star on all ten datasets, with no selection rule imposed. With few local epochs it stays near GcenG_{\rm cen}, the optimal Gram matrix of centralized training, and the UFM predicts where it ends in between.

The two idealizations (Appendix G.5 of the revised PDF).

  • Exact local optimization is replaced by EE gradient steps per round, with the bias trained together with the head (Theorem 3) and the features as well (Proposition 3), as in actual training. Appendix G.1–G.4 of the submission minimized both before every head step, which gave another small-budget end point, GwithinG_{\rm within} (the optimum for the within-client covariance).
  • The proximal selection rule is removed: the added analysis and the DNNs below use no proximal term. Q1 (reply 2/4) treats how gradient training selects a head without it.

Theory. In the fully active regime (every client's optimal Gram matrix is positive definite) with P≥CP\ge C, every client takes EE gradient steps with step sizes ηj>0\eta_j>0; put h=∑jηjh=\sum_j\eta_j and ε=h−∑jηj2/h\varepsilon=h-\sum_j\eta_j^2/h (constant steps: h=ηEh=\eta E, ε=η(E−1)\varepsilon=\eta(E-1)).

  • One step per round. A round is a gradient step on the averaged objective, so the centralized solution is fixed.
  • Small budgets. For small hh the rounds have a unique, locally attracting fixed point Gh=Gcen+εΔG+O(h2)G_h=G_{\rm cen}+\varepsilon\Delta_G+O(h^2), with ΔG\Delta_G in closed form from the target moments, for every feature learning rate and width.
  • Larger budgets. We compute the fixed points of the UFM dynamics numerically; as EE grows, the rounds approach the exact local optimization of Theorem 1, whose limit is G⋆G_\star.

DNNs: federated averaging by local SGD, no proximal term, no correction. Every run has 32,000 local epochs in total: R=⌊32000/E⌋R=\lfloor32000/E\rfloor rounds of EE epochs. The table gives the direction error dd (distance between Gram matrices scaled to unit norm) of the final Gram matrix to GcenG_{\rm cen} and G⋆G_\star, and the largest difference between its trace coordinate τ\tau (0 at GcenG_{\rm cen}, 1 at G⋆G_\star) and the UFM prediction τUFM\tau_{\rm UFM}. Ranges and "largest" are over the ten datasets; medians over three seeds (Supplementary Text S.12 in the supplementary material zip file).

EEroundsdd to GcenG_{\rm cen}dd to G⋆G_\starlargest ∣τ−τUFM∣\lvert\tau-\tau_{\rm UFM}\rvert
1320007.73×10−47.73\times10^{-4}–4.63×10−34.63\times10^{-3}2.59×10−12.59\times10^{-1}–6.46×10−16.46\times10^{-1}1.61×10−21.61\times10^{-2}
564001.28×10−31.28\times10^{-3}–1.32×10−21.32\times10^{-2}2.53×10−12.53\times10^{-1}–6.42×10−16.42\times10^{-1}1.83×10−21.83\times10^{-2}
506404.31×10−24.31\times10^{-2}–1.11×10−11.11\times10^{-1}2.01×10−12.01\times10^{-1}–5.99×10−15.99\times10^{-1}2.24×10−22.24\times10^{-2}
400802.45×10−12.45\times10^{-1}–5.66×10−15.66\times10^{-1}5.77×10−35.77\times10^{-3}–3.90×10−13.90\times10^{-1}1.89×10−21.89\times10^{-2}
400082.65×10−12.65\times10^{-1}–6.71×10−16.71\times10^{-1}3.02×10−33.02\times10^{-3}–3.21×10−23.21\times10^{-2}3.81×10−23.81\times10^{-2}
  • E=1E=1 and 5: the centralized solution. The final Gram matrix is within 1.32×10−21.32\times10^{-2} of GcenG_{\rm cen}, and at least 2.53×10−12.53\times10^{-1} from G⋆G_\star and 2.00×10−12.00\times10^{-1} from GwithinG_{\rm within}. Its small shift is that of Theorem 3: the first-order value τ1\tau_1 differs from τ\tau by at most 9.11×10−39.11\times10^{-3} on nine datasets (Swimmer: up to 2.25×10−22.25\times10^{-2}).
  • E=50E=50 and 400: the predicted path in between. The final Gram matrix lies between GcenG_{\rm cen} and G⋆G_\star, where the UFM places it: over the 110 dataset–budget cells of Figure 5, τ\tau differs from τUFM\tau_{\rm UFM} by 3.78×10−33.78\times10^{-3} in median and by at most 3.81×10−23.81\times10^{-2}. The only fitted number is the ratio r=10r=10 of the feature to the head learning rate, set once on the tabular runs with E≥50E\ge50.
  • E=4000E=4000: the BW barycenter. All ten datasets end closest to G⋆G_\star, within 3.21×10−23.21\times10^{-2}.

Scope. Private backbones with a shared head; the theory is for deterministic updates, the DNNs use local SGD, where EE counts epochs. The image and Swimmer runs at E=4000E=4000 were still moving toward G⋆G_\star. Full-model averaging (Table 8 in Appendix F.6) is outside the analysis.

The BW dynamics are thus the long-local-training end of actual federated averaging, and the UFM describes the other budgets.

AuthorsOpenReview ↗

Reply to Reviewer 6dL2 (2/4): Weakness, part 2, and Q1 (implicit selection without the proximal term)

Answer. Yes, approximately, and without error in one case: within the UFM framework of the paper, we identify the mechanism. The selection matters because a client's optimal head is unique only up to a rotation: all heads with one Gram matrix (its fiber) fit the client equally well, and averaging heads that differ by rotations shrinks the Gram matrix; the proximal term picks the head closest to the broadcast. Ordinary training adds no rotation of its own: with the bias and features at their optima before each head step (profiled updates), every gradient step is orthogonal to the fiber. The returned head is therefore determined by the path of its Gram matrix, and the closest head is the one reached without rotation along the straight path, the BW geodesic. When the broadcast Gram matrix commutes with the client covariance, the two paths lead to the same head: training returns the closest head without error. If all client covariances and this Gram matrix share one eigenbasis, as for scalar targets, the BW convergence result holds without the proximal penalty for profiled training run to convergence. Otherwise, near the client optimum, the two heads differ at most at second order in the distance of the broadcast from it; a round started at the barycenter G⋆G_\star then moves the Gram matrix only at third order in the client heterogeneity, while the trace of the gap to the centralized optimal Gram matrix GcenG_{\rm cen} is at least quadratic in it. These results partly explain why UFM simulations and trained DNNs without the proximal term both end near G⋆G_\star.

Theory (Appendix H.5 of the revised PDF; proofs in Supplementary Text S.10 in the supplementary material zip file), in the fully active UFM (P≥CP\ge C, client covariances above λHλWI\lambda_H\lambda_WI, broadcast Gram matrix G0≻0G_0\succ0).

  • Mechanism. Under profiled updates the gradient is −Ω(W)W-\Omega(W)W with Ω(W)\Omega(W) symmetric, hence orthogonal to the fiber directions W↦WQW\mapsto WQ; the returned head differs from the closest one by the rotation accumulated around the loop of the two Gram paths.
  • Theorem 7 (exact case). Under commutation this rotation vanishes: if G0Σ=ΣG0G_0\Sigma=\Sigma G_0, the profiled flow, and gradient descent with step size at most (λW+λmax⁡(Σ)/λH)−1(\lambda_W+\lambda_{\max}(\Sigma)/\lambda_H)^{-1}, converge to the closest optimal head. If all client covariances and G0G_0 share an eigenbasis, profiled training run to convergence reaches G⋆G_\star in one round.
  • Proposition 14 (general case). Without commutation the rotation is small near the client optimum: as δ=dBW(G0,GΣ)→0\delta=d_{\rm BW}(G_0,G_\Sigma)\to0, the limit of the profiled flow, or of profiled gradient descent with a small enough step size, is the closest optimal head plus an explicit term DW0DW_0 of order δ2\delta^2, plus O(δ3)O(\delta^3); DD depends on the step size and vanishes under commutation. In a UFM simulation the measured error is 0.943 to 0.999 times ∣DW0∣F\\|DW_0\\|_F.
  • Proposition 15 (cost over rounds). If every client runs complete profiled training from G⋆G_\star, then, as the heterogeneity β\beta of the client optima tends to 0, such a round lowers the trace by O(β4)O(\beta^4) and moves the Gram matrix by O(β3)O(\beta^3), whereas Tr(Gcen−G⋆)≥λmin⁡(G⋆)min⁡mpmβ2{\rm Tr}(G_{\rm cen}-G_\star)\ge\lambda_{\min}(G_\star)\min_mp_m\beta^2.
  • Proposition 16 (joint updates, first order). DNNs keep their features between rounds. The joint gradient flow, started near its minimizers with the kept features, returns, up to second-order terms, the optimal head closest to Wt+cmW~m,t−1W_t+c_m\widetilde W_{m,t-1} (of full row rank), where W~m,t−1\widetilde W_{m,t-1} is the client's previous returned head, cm=λW/(λHrm)c_m=\lambda_W/(\lambda_Hr_m) and rmr_m scales the feature steps (Appendix G.5). The closest heads satisfy this rule, and for equal clients, linearized at alignment, a misalignment shrinks by the factor c/(1+c)c/(1+c) per round.

UFM simulations and DNNs without the proximal term. Both end near G⋆G_\star.

  • Target moments of the ten datasets (β\beta from 3.29×10−13.29\times10^{-1} to 8.58×10−18.58\times10^{-1}). Rounds started at G⋆G_\star settle at a fixed Gram matrix within direction error (the distance between Gram matrices scaled to unit norm) 1.07×10−31.07\times10^{-3} of G⋆G_\star with profiled updates and 4.44×10−34.44\times10^{-3} with joint updates and kept features (rate ratio r=10r=10), whereas GcenG_{\rm cen} lies at 2.60×10−12.60\times10^{-1} to 6.42×10−16.42\times10^{-1}.
  • Main experiments (AdamW, E=4000E=4000 local epochs per round, 8 rounds; ten datasets, five seeds, 50 runs; Supplementary Text S.12). All 50 runs without the proximal term end closer in direction to G⋆G_\star than to GcenG_{\rm cen}; the median relative error of the final Gram matrix to G⋆G_\star is 1.63×10−21.63\times10^{-2}, against 2.90×10−42.90\times10^{-4} with the proximal weight ρ=10−3\rho=10^{-3} (Table 4 in Appendix F.1).

In short, the vanishing proximal penalty makes the selection exact; without it, the fiber mechanism makes profiled training perform it approximately in the UFM.

AuthorsOpenReview ↗

Reply to Reviewer 6dL2 (3/4): Q2 (how essential is the fully active regime?)

Reviewers LsbS (S2) and 7m7Y (Q1) raised a similar point.

Answer. The fully active regime is essential only for the uniqueness of the limit, in special configurations of the clients (Appendix H.1 of the revised PDF, within the UFM framework of the paper). A client is rank deficient when its targets vary too little in some direction (variance at most λHλW\lambda_H\lambda_W): the regularization then switches this direction off, and the client's optimal Gram matrix GmG_m is singular. Closest-head selection then still induces convergence to a BW barycenter of the GmG_m. This barycenter is unique if at least one client is fully active (keeps all directions) and, when none is, for almost every configuration of the clients; only in special configurations can the limit depend on the initial head. DNNs with rank-deficient clients end at the predicted limit on four of five datasets, and on all five with the correction.

What carries over (Appendix H.1; exact local optimization as in Theorem 1).

  • Client optimum and closest-head selection (Lemma 3). Gm=(αΣm1/2−λHI)+G_m=(\alpha\Sigma_m^{1/2}-\lambda_HI)_+ with α=λH/λW\alpha=\sqrt{\lambda_H/\lambda_W}, where (⋅)+(\cdot)_+ sets negative eigenvalues to zero: the formula of the submission, with the switched-off directions set to zero. The optimal head closest to the broadcast head is still unique at every round when the initial head has full row rank.
  • Convergence (Theorem 4). For arbitrary PSD GmG_m, every round decreases the BW objective V(X)=∑mpmdBW2(X,Gm)\mathcal V(X)=\sum_mp_md_{\rm BW}^2(X,G_m) of Theorem 1, and the shared Gram matrix converges to a minimizer of V\mathcal V, that is, to a BW barycenter. This limit can be singular.
  • Correction (Proposition 5). All corrected clients share the optimum (αΣcen1/2−λHI)+(\alpha\Sigma_{\rm cen}^{1/2}-\lambda_HI)_+, and one round returns it.

Is the limiting barycenter unique? (Theorem 5 in Appendix H.1) Call the range of GmG_m the support of client mm: the directions it keeps. V\mathcal V has a single minimizer (i) if at least one client is fully active (the submission assumed all); (ii) for almost every configuration of the supports, whatever the eigenvalues and the weights, even if all clients are rank deficient; (iii) if three supports are pairwise distinct hyperplanes.

Can the convergence behavior change? Outside these cases V\mathcal V can have several minimizers, and the limit can then depend on the initial head. Example: two clients that each keep one direction, orthogonal to each other. V\mathcal V fixes the diagonal of the limit but not its off-diagonal entry, which the initial head determines.

DNNs. In the main experiments (λH=λW=0.01\lambda_H=\lambda_W=0.01) every client is fully active, so we raised both penalties to λ=0.2\lambda=0.2 and 0.350.35 on the five image datasets; the second column counts the rank-deficient clients. The other columns give the direction error dd of the final Gram matrix to the predicted limit: the barycenter without the correction, the shared optimum of Proposition 5 with it. The barycenter is unique in every instance, by Theorem 5(i) where a client is fully active and by (iii) where none is. Local SGD, E=4000E=4000, eight rounds, proximal weight ρ=0.1λ\rho=0.1\lambda; medians over three seeds (Supplementary Text S.12 in the supplementary material zip file).

datasetrank-deficient clients at λ=0.2\lambda=0.2, 0.350.35no correction, λ=0.2\lambda=0.2no correction, λ=0.35\lambda=0.35with correction, λ=0.2\lambda=0.2with correction, λ=0.35\lambda=0.35
MPIIGaze6 of 8, 8 of 83.58×10−43.58\times10^{-4}1.23×10−41.23\times10^{-4}5.05×10−55.05\times10^{-5}7.09×10−57.09\times10^{-5}
300W-LP6 of 8, 7 of 88.98×10−48.98\times10^{-4}6.53×10−46.53\times10^{-4}4.84×10−54.84\times10^{-5}8.58×10−58.58\times10^{-5}
dSprites6 of 8, 8 of 88.04×10−48.04\times10^{-4}8.47×10−48.47\times10^{-4}3.44×10−53.44\times10^{-5}4.17×10−54.17\times10^{-5}
3D Shapes6 of 8, 8 of 87.15×10−47.15\times10^{-4}1.96×10−31.96\times10^{-3}4.73×10−54.73\times10^{-5}2.61×10−52.61\times10^{-5}
CIFAR-geo0 of 4, 4 of 44.68×10−34.68\times10^{-3}1.25×10−11.25\times10^{-1}–6.63×10−56.63\times10^{-5}
  • Without the correction. On the four eight-client datasets the DNNs end within 1.97×10−31.97\times10^{-3} of the barycenter at both penalties, as Theorem 4 predicts. Where the barycenter is singular (λ=0.35\lambda=0.35 on MPIIGaze, dSprites and 3D Shapes), their final Gram matrices are singular as well (smallest over largest eigenvalue at most 2.90×10−62.90\times10^{-6}). On CIFAR-geo the error is larger, an effect of the finite proximal weight: the UFM with the same weight gives nearly the same values, 5.17×10−35.17\times10^{-3} and 1.27×10−11.27\times10^{-1} (Supplementary Text S.6).
  • With the correction. The DNNs end within 8.58×10−58.58\times10^{-5} of the shared optimum in all ten corrected instances (the table and CIFAR-geo at λ=0.30\lambda=0.30), as Proposition 5 predicts.

Convergence thus holds without full activity, and the limit is unique outside special configurations of the supports.

Reviewer_LsbSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose a theoretical analysis of federated multivariate regression with private backbones and a shared linear head, using the unconstrained feature model (UFM). Because each client's optimal head is determined only up to an orthogonal transformation, they introduce a selection rule in which each client returns the optimal head closest to the broadcast head. They show that this rule arises as the limit of a vanishing FedProx-style proximal penalty. Under this rule, the shared head's Gram matrix follows a closed recursion that converges to the Bures–Wasserstein barycenter of the clients' optimal Gram matrices. The authors decompose the gap between this limit and the centralized optimum into three positive-semidefinite terms. These arise from differences in client target means, covariance heterogeneity, and the averaging of aligned heads. They also show that a correction based on a one-time exchange of target means and covariances recovers the centralized Gram matrix in a single round. UFM simulations and experiments with deep networks on five tabular and five image regression datasets show that the trained shared heads closely follow the predicted dynamics and endpoints, particularly with a weak proximal penalty.

Critical strengths
  • A clean and correct theoretical insight. The paper shows that averaging the optimal heads closest to the broadcast is exactly the Bures–Wasserstein barycenter iteration. This gives an exact limit for the shared Gram matrix. It also gives a three-term decomposition of the gap to centralized training: target-mean scatter, covariance heterogeneity, and head scatter. I checked the closest-head formula, the Gram recursion and the PSD terms, and they hold. This is the main reason I lean toward acceptance. It explains in an interpretable way why federated averaging still misses the centralized solution once the orthogonal ambiguity is removed.

  • Careful and honest experimental validation. The predicted Gram matrices are computed from target moments alone, independently of the trained networks, so the comparison is not circular. The authors test ten datasets with five seeds each and sweep both the local epochs and the proximal weight. They also try alignment instead of the proximal penalty, other architectures, and full-model averaging. They report where the agreement breaks down, such as at smaller local budgets. This matters to my assessment because it shows the UFM predictions carry over to trained networks in the stated regime, rather than resting only on simulations.

Critical weaknesses
  • The theory applies only in an extreme local-training regime, and its practical consequences are unclear. The analysis assumes every client reaches its local optimum in every round. The experiments approximate this with E=4000E = 4000 local epochs per round on 1,000–3,000 samples per client. At more realistic budgets the predictions break down. In Appendix F.3, the median direction error to G⋆G_\star is 0.46 at E=50E = 50 and 0.18 at E=400E = 400, and under full-model averaging at E=400E = 400 the endpoint is far from both G⋆G_\star and GcenG_{cen} (Table 8). Even within the regime, the predicted head Gram matrix has limited bearing on performance. The moment correction recovers GcenG_{cen} but not centralized predictions (Appendix C.3), and it lowers test MSE by only about 1.5% in median (Table 9). This is the main factor limiting my assessment: the paper offers a clean description of a limiting case, but it does not show what that description means for federated training as it is usually run.
Questions for authors

Clarification

  1. How do the predictions behave with few local epochs per round (for example E=1E = 1–55) and many more rounds? Appendix G suggests the endpoint lies near GwithinG_{within} for small local steps and moves toward G⋆G_\star as EE grows. Could you report endpoint errors to GwithinG_{within}, G⋆G_\star and GcenG_{cen} in that regime? Evidence that the characterization informs training at practical budgets would raise my assessment.

  2. Mismatch between the shared head and the private backbones. Evaluation pairs the final shared head with backbones that were trained against each client's own returned head. When MA≠0M_A \neq 0, these heads differ from the average. How much of the prediction error comes from this mismatch, and does it grow with client heterogeneity?

  3. Sharing target moments. The correction requires clients to upload their target means and covariances. Could you discuss the privacy implications, and how robust the correction is when these moments are noisy, for example under differential privacy?

Suggestions

  • Extend the analysis to partial participation and stochastic local updates.
  • Cover the rank-deficient regime that the fully active assumption excludes.
  • Evaluate test error with a non-oracle assignment of test samples to clients.
11 comments
AuthorsOpenReview ↗

Reply to Reviewer LsbS (1/8): Weakness and Q1, part 1 (exact local optimization; few local epochs)

We thank Reviewer LsbS for the very constructive comments. We answer them one by one below.

Reviewers 6dL2 (W) and 7m7Y (W1, Q3) raised a similar point.

Answer. We agree that the submission described only the limit of exact local optimization. Within the UFM framework of the paper, we extended the analysis to any local budget EE at a fixed total budget, and ran the sweep asked for in Q1: EE from 1 to 4000 on ten datasets. With few local epochs the final Gram matrix stays at the centralized GcenG_{\rm cen}; as EE grows it moves to the barycenter G⋆G_\star of the submission, and the UFM predicts where it is at each budget. The errors to G⋆G_\star at E=50E=50 and 400400 that the reviewer cites are intermediate points of this path, which the UFM reproduces in the sweep.

Small budgets: GcenG_{\rm cen}, not GwithinG_{\rm within}. Appendix G.1–G.4 of the submission pointed to GwithinG_{\rm within} because it minimized the features and the bias before every head step. In actual training they are trained by gradient steps together with the head. The added analysis (Appendix G.5 of the revised PDF) makes only this change (bias: Theorem 3; features as well: Proposition 3), and the small-budget end point becomes GcenG_{\rm cen}.

Theory (Appendix G.5). In the fully active UFM with P≥CP\ge C, every client takes EE local gradient steps with step sizes ηj>0\eta_j>0; put h=∑jηjh=\sum_j\eta_j and ε=h−∑jηj2/h\varepsilon=h-\sum_j\eta_j^2/h (constant steps: h=ηEh=\eta E, ε=η(E−1)\varepsilon=\eta(E-1)).

  • One step per round. A round is a gradient step on the averaged objective, and the centralized state is fixed.
  • Small budgets (Theorem 3 and Proposition 3 in Appendix G.5). For small hh the rounds have a unique, locally attracting fixed point Gh=Gcen+εΔG+O(h2)G_h=G_{\rm cen}+\varepsilon\Delta_G+O(h^2), with ΔG\Delta_G in closed form from the target moments, for every feature learning rate and width.
  • Larger budgets. We compute the fixed points of the UFM dynamics numerically; as EE grows, the rounds approach the exact local optimization of Theorem 1, whose limit is G⋆G_\star.

DNNs: the local-budget sweep. Ordinary training by local SGD, whose learning rates are the step sizes ηj\eta_j, with 32,000 local epochs in total: R=⌊32000/E⌋R=\lfloor32000/E\rfloor rounds of EE epochs. The table gives the direction error dd of the final Gram matrix to the three matrices named in Q1, and the largest difference between its trace coordinate τ\tau (0 at GcenG_{\rm cen}, 1 at G⋆G_\star) and the prediction τUFM\tau_{\rm UFM} of the numerical fixed point. Ranges and "largest" are over the ten datasets; medians over three seeds (Supplementary Text S.12 in the supplementary material zip file).

EEroundsdd to GcenG_{\rm cen}dd to GwithinG_{\rm within}dd to G⋆G_\starlargest ∣τ−τUFM∣\lvert\tau-\tau_{\rm UFM}\rvert
1320007.73×10−47.73\times10^{-4}–4.63×10−34.63\times10^{-3}2.06×10−12.06\times10^{-1}–6.06×10−16.06\times10^{-1}2.59×10−12.59\times10^{-1}–6.46×10−16.46\times10^{-1}1.61×10−21.61\times10^{-2}
564001.28×10−31.28\times10^{-3}–1.32×10−21.32\times10^{-2}2.00×10−12.00\times10^{-1}–6.02×10−16.02\times10^{-1}2.53×10−12.53\times10^{-1}–6.42×10−16.42\times10^{-1}1.83×10−21.83\times10^{-2}
506404.31×10−24.31\times10^{-2}–1.11×10−11.11\times10^{-1}1.50×10−11.50\times10^{-1}–5.59×10−15.59\times10^{-1}2.01×10−12.01\times10^{-1}–5.99×10−15.99\times10^{-1}2.24×10−22.24\times10^{-2}
400802.45×10−12.45\times10^{-1}–5.66×10−15.66\times10^{-1}9.36×10−39.36\times10^{-3}–3.47×10−13.47\times10^{-1}5.77×10−35.77\times10^{-3}–3.90×10−13.90\times10^{-1}1.89×10−21.89\times10^{-2}
400082.65×10−12.65\times10^{-1}–6.71×10−16.71\times10^{-1}3.88×10−23.88\times10^{-2}–1.86×10−11.86\times10^{-1}3.02×10−33.02\times10^{-3}–3.21×10−23.21\times10^{-2}3.81×10−23.81\times10^{-2}
  • E=1E=1 and 5 (Q1). The final Gram matrix is within 1.32×10−21.32\times10^{-2} of GcenG_{\rm cen}, and at least 2.00×10−12.00\times10^{-1} from GwithinG_{\rm within} and 2.53×10−12.53\times10^{-1} from G⋆G_\star: training ends at the centralized solution. The small shift is that of Theorem 3: its first-order value τ1\tau_1 differs from τ\tau by at most 9.11×10−39.11\times10^{-3} on nine datasets (Swimmer: up to 2.25×10−22.25\times10^{-2}).
  • E=50E=50 and 400. The final Gram matrix lies between GcenG_{\rm cen} and G⋆G_\star, hence the large errors to G⋆G_\star there. The UFM gives its position: over the 110 dataset–budget cells of Figure 5, τ\tau differs from τUFM\tau_{\rm UFM} by 3.78×10−33.78\times10^{-3} in median and by at most 3.81×10−23.81\times10^{-2}. The only fitted number is the ratio r=10r=10 of the feature to the head learning rate, set once on the tabular runs with E≥50E\ge50.
  • E=4000E=4000. All ten datasets end closest to G⋆G_\star, as in the submission.

Scope. Private backbones with a shared head; the theory is for deterministic updates, the DNNs use local SGD, where EE counts epochs. The image and Swimmer runs at E=4000E=4000 were still moving toward G⋆G_\star. Full-model averaging (Table 8) is outside the analysis.

The framework thus covers all local budgets at a fixed total budget, with the exact local optimization of the submission as its large-EE end.

AuthorsOpenReview ↗

Reply to Reviewer LsbS (2/8): Weakness and Q1, part 2 (bearing on performance: Appendix C.3 and the 1.5% of Table 9)

Reviewer 6dL2 (Q4) raised a similar point.

Answer. We agree with both observations, and the revision explains them. In the UFM, the training predictions are fixed by the head Gram matrix together with two biases: the one around which each client fits its features, and the one used to predict. The paper's correction CmC_m sets the Gram matrix to GcenG_{\rm cen} but leaves the two biases different; this is the gap of Appendix C.3 and the reason for the gain of only 1.5%. Fixing the bias at the global target mean and adapting the correction removes the gap: the training predictions become the centralized ones, and in DNNs the test error on the clients' own data is 81.4% below ordinary training in the statistic of Table 9, where CmC_m gives 1.5%. The Gram matrix alone is thus not sufficient for the predictions, but it is needed: with the bias fixed and no correction, this test error at E=4000E=4000 stays 5.36–49.5% above that of ordinary training at E=5E=5, and with the adapted correction it is within 5% (four tabular datasets, reply 3/8).

Why CmC_m does not recover the predictions (Appendix H.2 of the revised PDF). In the UFM, let client mm hold the features that are optimal for a head WW and a bias cmc_m, and let the shared model predict with WW and a bias bb. With YmY_m the targets of client mm, the training predictions are

Y^m=GA−1(Ym−cm1⊤)+b1⊤,G=WW⊤,A=G+λHI.\widehat Y_m=GA^{-1}(Y_m-c_m\mathbf1^\top)+b\mathbf1^\top,\qquad G=WW^\top,\quad A=G+\lambda_HI.

They depend only on the Gram matrix GG and on the two biases cmc_m and bb. Centralized training has G=GcenG=G_{\rm cen} and cm=b=μgc_m=b=\mu_g, the global target mean, and its training MSE is λHλW\lambda_H\lambda_W. The correction CmC_m also gives G=GcenG=G_{\rm cen}, but cm=μmc_m=\mu_m, the target mean of client mm, while b=μgb=\mu_g. The training MSE is then (Proposition 7(2) in Appendix H.2)

λHλW+1CTr[(I−λHλWΣcen−1)Σμ],Σμ=∑mpm(μm−μg)(μm−μg)⊤.\lambda_H\lambda_W+\tfrac1C\mathrm{Tr}\bigl[(I-\lambda_H\lambda_W\Sigma_{\rm cen}^{-1})\Sigma_\mu\bigr],\qquad\Sigma_\mu=\sum_mp_m(\mu_m-\mu_g)(\mu_m-\mu_g)^\top.

The second term is the gap of Appendix C.3 written as a training error. It is positive whenever the client means differ, even though the Gram matrix is exactly GcenG_{\rm cen}.

How to remove it (Proposition 4 in Appendix G.5; Proposition 7(3) in Appendix H.2). Fix the bias of all clients at μg\mu_g and replace CmC_m by Cm′(W)=λH2Tr{(Σcen−Sm(μg))A−1}C_m'(W)=\frac{\lambda_H}{2}\mathrm{Tr}\lbrace(\Sigma_{\rm cen}-S_m(\mu_g))A^{-1}\rbrace, where Sm(μg)=Σm+(μm−μg)(μm−μg)⊤S_m(\mu_g)=\Sigma_m+(\mu_m-\mu_g)(\mu_m-\mu_g)^\top is the second moment of client mm's targets about μg\mu_g; it needs the same moments as CmC_m. Now G=GcenG=G_{\rm cen} and cm=b=μgc_m=b=\mu_g, as in centralized training: the training predictions are the centralized ones, and the training MSE is λHλW\lambda_H\lambda_W.

DNNs (protocol of Table 9: AdamW, E=4000E=4000, eight rounds). The table gives the change of the test MSE against ordinary training on the same seed, for CmC_m (median over eight seeds) and for the bias fixed at μg\mu_g with Cm′C_m' (three seeds; ρ=10−3\rho=10^{-3}); the runs are listed in Supplementary Text S.12 in the supplementary material zip file. A negative value means a lower error than ordinary training. "Own data": each test sample is predicted by the client to which Table 9 assigns it (by its target range; at random on CIFAR-geo). "Ensemble": by the pmp_m-weighted average of the predictions of all clients.

datasetown data: CmC_mown data: fixed bias, Cm′C_m'ensemble: CmC_mensemble: fixed bias, Cm′C_m'
CIFAR-geo-1.33%-13.2%-12.6%-8.09%
dSprites-2.16%-97.8%-1.70%-0.893%
3D Shapes-2.02%-99.9%-17.7%-10.5%
MPIIGaze-0.0618%-62.8%-0.488%-1.36%
300W-LP-1.44%-81.4%-1.50%-1.09%
  • Training. With the fixed bias and Cm′C_m', the training MSE of the shared model is 1.00–1.14 times λHλW\lambda_H\lambda_W on all ten datasets: the DNNs reach the centralized training error, as Proposition 7(3) predicts.
  • Test, own data. With CmC_m the error is at most 2.16% below ordinary training; with the fixed bias and Cm′C_m' it is 13.2% to 99.9% below. In the statistic of Table 9, the median over the 15 dataset–seed combinations, this is 81.4% against 1.5%. Both corrections aim at GcenG_{\rm cen} and differ in the bias, so the gain of Table 9 was limited by the trained shared bias.
  • Test, ensemble. Both corrections lower the error on all five datasets. Compared on the same seeds, the fixed bias is below CmC_m on MPIIGaze only (-0.719% to +7.81%): what it adds to CmC_m is in the predictions of each client on its own data.

Scope. The comparison with CmC_m changes the bias and the correction together, and that of reply 3/8 adds the proximal term together with Cm′C_m'. The assignment of Table 9 uses the label on four datasets; reply 8/8 gives the results without the label. The UFM describes the features of the training samples only, so the test errors are measurements.

AuthorsOpenReview ↗

Reply to Reviewer LsbS (3/8): Weakness and Q1, part 3 (training as it is usually run)

Reviewers 6dL2 (W) and 7m7Y (W1, Q3) raised a similar point.

Answer. For training as it is usually run, the characterization says what long local training costs and gives two ways to act on it; we tested both on DNNs over the sweep of reply 1/8. Under ordinary averaging, long local training raises the own-data test error (each test sample predicted by its own client). (i) With the bias fixed and the correction adapted to it, the error stays within 5% of its level at short local training at every budget, so 8 rounds at E=4000E=4000 match 6400 at E=5E=5. (ii) Without a correction, the UFM gives, before training, the largest EE that keeps the end point near the centralized solution.

Why long local budgets, and why the bias. Long local budgets are chosen when devices can afford many local epochs but communication is scarce, which motivated federated averaging (McMahan et al., 2017). On heterogeneous data they cause client drift (Li et al., 2020; Wang et al., 2020). In the UFM, ordinary training then moves the Gram matrix from GcenG_{\rm cen} to G⋆G_\star (reply 1/8); without the proximal term, profiled local training returns, near the client optimum, the closest optimal head up to an error of second order in the distance of the broadcast from it (Appendix H.5 of the revised PDF), which helps explain why ordinary training follows the theory. The shared bias adds an offset to the error (Lemma 5 in Appendix H.2; reply 4/8). With the bias fixed at μg\mu_g and Cm′C_m' (reply 2/8), the centralized state is fixed at every local budget (Proposition 4 in Appendix G.5).

DNNs (the sweep of reply 1/8: local SGD, 32,000 local epochs in total). The table gives the own-data test MSE at three budgets, for ordinary training and for the fixed bias with Cm′C_m': four tabular datasets, MSE per target coordinate, medians over three seeds (Supplementary Text S.12 in the supplementary material zip file); the fixed-bias runs carry ρ=10−3\rho=10^{-3}.

datasetE=5E=5 (6400 rounds): ordinaryE=5E=5: fixed bias + Cm′C_m'E=50E=50 (640 rounds): ordinaryE=50E=50: fixed bias + Cm′C_m'E=4000E=4000 (8 rounds): ordinaryE=4000E=4000: fixed bias + Cm′C_m'
Walker5.91×10−15.91\times10^{-1}5.93×10−15.93\times10^{-1}5.92×10−15.92\times10^{-1}5.86×10−15.86\times10^{-1}9.19×10−19.19\times10^{-1}5.87×10−15.87\times10^{-1}
HalfCheetah6.96×10−26.96\times10^{-2}6.98×10−26.98\times10^{-2}7.89×10−27.89\times10^{-2}6.99×10−26.99\times10^{-2}4.73×10−14.73\times10^{-1}6.96×10−26.96\times10^{-2}
Hopper1.12×10−11.12\times10^{-1}1.14×10−11.14\times10^{-1}1.29×10−11.29\times10^{-1}1.14×10−11.14\times10^{-1}7.85×10−17.85\times10^{-1}1.17×10−11.17\times10^{-1}
Beijing1.59×10−11.59\times10^{-1}1.62×10−11.62\times10^{-1}2.22×10−12.22\times10^{-1}1.61×10−11.61\times10^{-1}8.31×10−18.31\times10^{-1}1.63×10−11.63\times10^{-1}
  • Ordinary training. The error grows with the budget: at E=4000E=4000 it is 1.55–7.02 times its value at E=5E=5.
  • Fixed bias alone (no Cm′C_m', no proximal term; not in the table). At E=4000E=4000 the error is 32.1–79.4% below ordinary training but still 5.36–49.5% above the E=5E=5 level: the shared bias causes most of the rise, but not all of it.
  • Fixed bias with Cm′C_m'. The error stays within 5% of ordinary training at E=5E=5 at all eleven budgets (-3.20% to +4.68% over the 44 dataset–budget cells). The Gram matrix stays at GcenG_{\rm cen} as Proposition 4 predicts (direction error at most 8.65×10−48.65\times10^{-4}; training MSE 1.00–2.81 times λHλW\lambda_H\lambda_W).
  • Image datasets (Cm′C_m' at E=1E=1, 5, 160, 400). The Gram matrix stays within 1.13×10−31.13\times10^{-3} of GcenG_{\rm cen}, and at E=160E=160 and 400400 the own-data test error is -11.9% to +2.39% from ordinary training at E=5E=5.
  • Where the gain is. With the pmp_m-weighted ensemble of all clients, Cm′C_m' is below ordinary training on 1 of 4 tabular datasets at E=4000E=4000, and above it on 3 of 5 image datasets at E=160E=160 and 400400: the gain is in the own-data predictions.

Choosing EE without a correction. A larger EE saves rounds but moves the end point away from GcenG_{\rm cen} and raises the test error. The UFM tells before training how large EE may be: its fixed points need only the target moments, learning rates and r=10r=10 (reply 1/8), so one takes the largest EE whose fixed point is within a chosen tolerance of GcenG_{\rm cen}. For a tolerance of 0.01 in direction error (peak learning rate 0.03), the fixed point is within 7.39×10−37.39\times10^{-3} of GcenG_{\rm cen} at E=16E=16 and at least 1.35×10−21.35\times10^{-2} away at E=50E=50, so E=16E=16 is chosen. DNNs trained at both budgets end 3.58×10−33.58\times10^{-3}–6.48×10−36.48\times10^{-3} from GcenG_{\rm cen} at E=16E=16, inside the tolerance, and 1.18×10−21.18\times10^{-2}–2.11×10−22.11\times10^{-2} at E=50E=50, outside it, as predicted (four datasets, seeds {5,6,7}; Appendix G.5).

Scope. Proposition 4 concerns deterministic updates; the numbers show how closely the SGD DNNs follow it. The UFM gives no guarantee for test error.

In these two ways, the characterization informs training at practical budgets.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the rebuttal

We sincerely thank the AC and all reviewers for their time and constructive comments.

In the initial reviews, all three reviewers gave the submission a positive score (3). In the strengths, the reviewers recognized the theoretical analysis (the ambiguity of locally optimal heads, the Bures–Wasserstein limit of the shared Gram matrix and the decomposition of the gap to centralized training) and its validation with DNNs on ten datasets. Following their constructive comments, we clarified and supplemented the paper within the same theoretical framework, without changing its main results. The concerns and the replies that address them are as follows.

  • Exact local optimization (all three reviewers). With few local epochs, training ends near the centralized GcenG_{\rm cen} (Theorem 3 and Proposition 3 in Appendix G.5), and as EE grows it moves toward the barycenter G⋆G_\star; DNNs on ten datasets with EE from 1 to 4000 follow the UFM prediction (Section 5.3, Figure 5). The framework thus covers the whole range of local budgets, including the small budgets closest to practice. Replies: LsbS 1/8, 6dL2 1/4, 7m7Y 1/5.

  • Rank-deficient clients (all three reviewers). The fully active regime matters only for the uniqueness of the limit. When a client's targets vary too little in some direction, the regularization switches this direction off and the client's optimal Gram matrix is singular; the closest-head selection then still converges to a BW barycenter, and the correction still applies. This barycenter is unique if at least one client is fully active and, otherwise, for almost every configuration of the clients (Appendix H.1). DNNs with rank-deficient clients end at the predicted limit on four of five datasets, and on all five with the correction. Replies: LsbS 7/8, 6dL2 3/4, 7m7Y 3/5.

  • The proximal selection rule. Without an explicit proximal term, which is closer to federated training in practice, DNNs still end near G⋆G_\star at E=4000E=4000 on all ten datasets. In the UFM, ordinary local training with the bias and features at their optima performs the closest-head selection approximately near the client optimum, through the fiber mechanism (Appendix H.5), which helps explain this. Replies: 6dL2 2/4, 7m7Y 1/5, LsbS 3/8.

  • Predictions and test error. Appendix H.2 relates the Gram matrix to the predictions of the shared model: in the UFM, using the adapted correction Cm′C_m' with a fixed bias recovers the centralized training predictions. In DNNs, it lowers the own-data test error by 81.4% relative to ordinary training on the five image datasets, in the median statistic of Table 9. This substantial improvement offers insight for the design of practical federated algorithms.

    The mismatch between the shared head and the private backbones is a small part of the prediction error, and without the oracle assignment of test samples both corrections still lower the ensemble test error on all five image datasets under the protocol of Table 9 (Appendix H.2). Replies: LsbS 2/8 to 4/8 and 8/8, 6dL2 4/4.

  • Partial participation. In the theory, with the correction the shared Gram matrix equals GcenG_{\rm cen} for every set of participants; without it, under random participation of equally weighted clients, it fluctuates around G⋆G_\star by an amount that the theory gives (Appendix H.3). DNNs with several numbers of participants per round follow both results. Replies: LsbS 6/8, 7m7Y 2/5.

  • The moment correction in practice: cost, privacy, noise and small samples. Each client sends its moments only once; this message and the computation that the correction adds to each local update grow with the number of outputs, not with the number of clients. The moments can be released once with a differential-privacy guarantee, and the correction stays effective under the added noise, since an error common to all clients moves its end point at most linearly. The inaccurate moments of small clients do not change the target of the correction, because all moments are pooled through exact sums (Appendix H.4). Replies: LsbS 5/8; 7m7Y 4/5, 5/5 and the follow-up reply.

All additions to the paper are marked in blue in the revised PDF; the Supplementary Text in the supplementary material zip file gives their complete statements, the key steps of every proof and the DNN details.

We thank the AC and all reviewers once again for their careful reading and constructive comments, which have helped us improve the paper.

Best regards,
The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.