- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies ambiguity in federated multivariate regression with private backbones and a shared linear head, using an unconstrained feature model (UFM) that treats training-sample features as free variables. The authors propose a natural solution selection rule and show that the shared Gram matrix converges to the Bures–Wasserstein barycenter of clients’ optimal Gram matrices. This paper further analyzehe the gap to centralized training and its one-round correction from shared target moments.
Critical strengths
1.This paper derives a closed recursion for the shared Gram matrix and proves that it converges to the unique Bures–Wasserstein barycenter of the clients’ optimal Gram matrices under the positive definite assumptions. 2.The authors verify their results numerically in the UFM and test predictions on five tabular and five image regression datasets using deep networks with feature regularization and long local training.
Critical weaknesses
1.The main results assume that clients reach their local optima in every round. The assumption differ substantially from practical federated learning, where local training is incomplete, participation is intermittent. 2.The correction requires a one-time exchange of the clients’ target means and covariances. However, the paper does not sufficiently establish the communication cost, scalability, or privacy implications of sharing these statistics in practical deployments.
Questions for authors
1.The analysis relies on positive definite optimal Gram matrices. Can the convergence and selection results be extended to positive-semidefinite or rank-deficient Gram matrices? 2.What are the communication and computational costs of exchanging target means and covariances, particularly when the output dimension is large or the number of clients increases? Could the authors further discuss practical strategies to mitigate these coosts and evaluate the scalability of the proposed correction under such settings? 3.The authots assume that clients reach their local optima in every round , which may not always hold in practical federated learning systems. Could the authors discuss how violations of this assumption may affect the proposed method and whether the main conclusions can be expected to hold under more realistic training settings? 4.The correction relies on the target means and covariance matrices estimated from local data. In practice, these estimates may be unreliable when some clients have only a limited number of training samples. How would such estimation errors affect the effectiveness of the proposed correction, particularly under data-imbalanced settings?
9 comments
Reply to Reviewer 7m7Y (2/5): Weakness 1, second part (intermittent participation)
Reviewer LsbS (S1) raised a similar point.
Answer. With the correction, the theory shows that intermittent participation does not move the Gram matrix: it is the centralized after every round, for any pattern of participating clients, including clients that are absent for several rounds. Without the correction, random participation keeps the Gram matrix fluctuating; for equally weighted clients it stays around the barycenter , at a distance set by two factors: one decreases with the number of participants and is 0 when all clients take part, and the other grows with the heterogeneity of the clients. Within the UFM framework of the paper, we extended the results to rounds in which only some clients participate, and trained DNNs with several numbers of participants per round; the DNNs follow both results.
Theory (Appendix H.3 of the revised PDF; proofs in Supplementary Text S.8 in the supplementary material zip file). The theory assumes exact local optimization and closest-head selection, as in Theorem 1. It treats the rounds with the correction first, and then the rounds without it.
- Proposition 8 (with the correction). Absent clients do not affect the corrected rounds. The reason is that the paper's correction uses the moments of all clients, exchanged once, so clients that have not yet participated are included. All participants of a round then minimize the same objective from the same broadcast and return the unique optimal head closest to this broadcast, so for every and for any nonempty sets of participants. The shared bias follows the participants' target means; fixing it at the global target mean , with the adapted correction of reply 1/5, keeps head and bias both fixed.
- Theorem 6 and Proposition 9 (without the correction). Each set of participants now has its own barycenter, and a round moves the Gram matrix toward the barycenter of its participants. If clients are drawn independently and uniformly in each round and the client Gram matrices differ, this target keeps changing, so has almost surely no limit. The size of its motion around can still be computed. For equal weights and clients with close Gram matrices, the long-run mean-square BW distance to is
up to higher-order terms: a sampling factor that is 0 at full participation, times the head scatter of the clients' optimal Gram matrices .
DNNs. The DNNs agree with both parts of the theory. With the correction and half of the clients per round, the final Gram matrix is within of (relative Frobenius error) in all 21 runs on seven datasets. Without it, the seed-median mean-square BW distance to over rounds 9–16 is 0.760 to 1.21 times the formula evaluated on each run's moments, and 0.672 to 1.22 times for 16 further combinations of dataset and ; a factor of 1 would be exact agreement. Protocol: AdamW, local epochs per round, 16 rounds, proximal weight , three seeds (Supplementary Text S.12).
In short, the correction removes the effect of intermittent participation on the Gram matrix, exactly in the UFM and to within the error above in the DNNs. Without the correction, this effect has the size given by the formula above.
Reply to Reviewer 7m7Y (3/5): Q1 (positive-semidefinite and rank-deficient Gram matrices)
Reviewers LsbS (S2) and 6dL2 (Q2) raised a similar point.
Answer. Yes, in the theory both results extend to arbitrary positive-semidefinite Gram matrices (Appendix H.1 of the revised PDF, within the UFM framework of the paper). A client is called fully active when its optimal Gram matrix is positive definite, and rank deficient when is singular: its targets then vary too little in some direction (variance at most ), and the regularization switches this direction off. Selection: the closest optimal head is still unique at every round, from a full-row-rank initial head. Convergence: the shared Gram matrix still converges to a minimizer of the same BW objective. Full activity made this minimizer unique; with rank-deficient clients it is unique outside special configurations. DNNs with rank-deficient clients end at the predicted limit on four of five datasets, and on all five with the correction.
The selection result (Lemma 3 in Appendix H.1). The client optimum is with , where sets negative eigenvalues to zero: the formula of the submission with the switched-off directions set to zero. The optimal head closest to a broadcast head is unique if and only if is positive definite on the directions that the client keeps, and it then has a closed form, as in Lemma 1. With an initial head of full row rank, this holds at every round.
The convergence result (Theorem 4 in Appendix H.1). For arbitrary positive-semidefinite and an initial head of full row rank, every round decreases the BW objective of Theorem 1, and the shared Gram matrix converges to a minimizer of . This limit can be singular.
What changes: uniqueness of the limit (Theorem 5 in Appendix H.1). With positive-definite , has a single minimizer, the barycenter . With rank-deficient it can have several, and the limit can then depend on the initial head. Call the range of the support of client : the directions it keeps. The minimizer is unique (i) if at least one client is fully active (the submission assumed all); (ii) for almost every configuration of the supports, whatever the eigenvalues and the weights, even if every is rank deficient; (iii) if three supports are pairwise distinct hyperplanes. Example: two clients that each keep one direction, orthogonal to each other; then fixes the diagonal of the limit but not its off-diagonal entry, which the initial head determines.
DNNs. In the main experiments () every client is fully active, so we raised both penalties to and on the five image datasets; the second column counts the rank-deficient clients. The other columns give the direction error of the final Gram matrix to the predicted limit: the minimizer of without the correction; with it, the optimum shared by all corrected clients, which one round still returns (Proposition 5 in Appendix H.1). The minimizer is unique in every instance, by Theorem 5(i) where a client is fully active and by (iii) where none is. Local SGD, , eight rounds, proximal weight ; medians over three seeds (Supplementary Text S.12 in the supplementary material zip file).
| dataset | rank-deficient clients at , | no correction, | no correction, | with correction, | with correction, |
|---|---|---|---|---|---|
| MPIIGaze | 6 of 8, 8 of 8 | ||||
| 300W-LP | 6 of 8, 7 of 8 | ||||
| dSprites | 6 of 8, 8 of 8 | ||||
| 3D Shapes | 6 of 8, 8 of 8 | ||||
| CIFAR-geo | 0 of 4, 4 of 4 | – |
- Without the correction. On the four eight-client datasets the final Gram matrix is within of the minimizer at both penalties. Where the minimizer is singular ( on MPIIGaze, dSprites and 3D Shapes), the final Gram matrices are singular as well (smallest over largest eigenvalue at most ). CIFAR-geo ends farther away because of the finite proximal weight: with the same weight the UFM gives and (Supplementary Text S.6).
- With the correction. All ten corrected instances (the table and CIFAR-geo at ) end within of the shared optimum.
Scope. The theorems are for exact local optimization with closest-head selection.
Reply to Reviewer 7m7Y (1/5): Weakness 1, first part, and Q3 (incomplete local training)
We thank Reviewer 7m7Y for the very constructive comments. We answer them one by one below.
Reviewers LsbS (W, Q1, S1) and 6dL2 (W) raised a similar point.
Answer. Within the UFM framework of the paper, we extended the analysis from local optima in every round to any number E of local gradient steps per round, and trained DNNs with E from 1 to 4000 local epochs on ten datasets. To the two parts of Q3:
Theory (Appendix G.5). Every client takes E gradient steps of size η per round on the head and the bias; client optima are positive definite, as in the submission.
DNNs. Local SGD with E epochs per round and 32,000 local epochs in total; medians over three seeds (Supplementary Text S.12 in the supplementary material zip file). The table gives, for ordinary training, the direction error d (the distance between Gram matrices scaled to unit norm) of the final Gram matrix, as ranges over the ten datasets.
Scope. Private backbones with a shared head; the theory is for deterministic updates, the DNNs use local SGD. The image and Swimmer runs at E=4000 were still moving toward G⋆.