acceptodds
Under review as a conference paper at ICLR 2027

Averaging Models Is Not Averaging Spectra: Steering Federated Representations

Abstract

Federated learning averages model parameters, whereas representation diversity belongs to the features produced by the aggregated model. We study the gap between these two objects through class-conditional von Neumann entropy (VNE). Exact linear constructions reveal a two-sided phenomenon: parameter averaging can either increase or decrease VNE relative to the local encoders. This spectral ambiguity leads to a second question—how far can the aggregated representation depart from centralized training? Under explicit regularity and heterogeneity assumptions, we bound the one-round diagnostic deviation from a same-start centralized reference, including rank-deficient empirical covariances. We then turn from diagnosis to control: interpolation toward a shared class-conditional density matrix contracts a weighted spectral disagreement, motivating the class-conditional FedVNA regularizer. On CIFAR-10 with ResNet-18 and Dirichlet , FedVNA raises final VNE from FedAvg's to , while their best accuracies remain close ( and ). Together, the results show that federated spectra can be diagnosed and deliberately shaped, while spectral diversity and predictive performance remain distinct outcomes.

Primary area:
learning theory
Keywords:
federated learning, representation learning, spectral diversity, von Neumann entropy, non-IID data, class-conditional regularization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SbXfOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how parameter averaging affects class-conditional representation entropy. It provides linear examples with opposite entropy changes, bounds one-round deviation from a same-start centralized reference, and establishes disagreement contraction along an ideal density-matrix interpolation. It then proposes FedVNA, combining a shared class-conditional reference with an entropy regularizer.

Critical strengths
  1. The analysis distinguishes the covariance of an averaged encoder from averaged client covariances, with explicit examples showing that aggregation can either increase or decrease VNE.

  2. The experiments show spectral changes alongside largely unchanged accuracy, and report seed variation and the added computational and communication costs.

Critical weaknesses
  1. The covariance decomposition, entropy counterexamples, and interpolation contraction rely largely on standard identities and convexity properties. The additional insight beyond existing VNE regularization and federated representation decorrelation remains limited.

  2. Figure 4 states that zero-alignment settings use 64-dimensional features, whereas nonzero-alignment settings use 512 dimensions. The sweep therefore does not isolate the alignment term at a fixed feature dimension.

  3. The experiments do not directly measure aggregation-induced entropy changes on fixed diagnostic samples or deviations from the same-start centralized reference. The practical informativeness of the theoretical analysis is therefore unclear.

  4. FedVNA increases round time from about 25 seconds to 91–105 seconds and adds 10 MiB of communication per client per round in each direction, while accuracy remains close to FedAvg. Evidence is limited to CIFAR-10 with ResNet-18.

  5. Clients with insufficient samples of a class contribute a maximally mixed matrix to its reference. Under strong label skew, this replaces missing information with isotropic covariance, but its effect on the reference and disagreement metric is not evaluated.

  6. The training surrogate uses a ridge-regularized Gram matrix, whereas the theoretical diagnostic uses an unregularized normalized covariance. The regularized entropy can be maximal even for zero centered features, and the experiments do not assess sensitivity to ridge strength or feature scale.

Questions for authors
  1. What does the class-conditional analysis establish beyond existing VNE regularization and federated representation decorrelation?

  2. Why do the zero-alignment settings use 64-dimensional features while the nonzero-alignment settings use 512 dimensions? Are there existing comparisons at the same dimension that isolate the alignment term?

  3. Do any existing results measure local-model and aggregated-model VNE on the same fixed diagnostic samples, or compare the aggregated model with the same-start centralized reference?

  4. Given the reported computational and communication overhead, what benefit does FedVNA provide beyond increasing the entropy metric that it directly regularizes?

  5. Under each heterogeneity level, what fraction of the class-reference weight comes from maximally mixed fallback matrices? How sensitive are the conclusions to this treatment of clients with insufficient class samples?

  6. What weights qk,yq_{k,y} are used to aggregate the client–class diagnostic entropies, and how are pairs with insufficient samples or zero covariance handled?

  7. Are uploaded class covariances recomputed using the final fixed local model, or accumulated from features produced by changing models during the final local epoch?

Reviewer_texmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper attempts to study the difference between averaging model parameters vs averaging their representations. I think this is the point, but honestly, I am not sure, because it seems almost entirely AI generated. This is ethically fine, because the authors have acknowledged AI use. However, it is hard to understand the model, results or the significance of the paper. More details in the weakness comments below.

Critical strengths

It is hard for me to evaluate the strengths as I am unable to understand the message (even though my research is in related areas).

Critical weaknesses

Starting from the first line, I think that there are issues. “FedAvg (McMahan et al., 2017) combines client parameters, but a representation spectrum is observed only after the resulting model acts on data.” – Yes, FedAvg updates the model weights, but what representation is alluded to here? On reading further, this is unclear to me. Is there a linear model (or some other model) of the final representation, the spectra is this resulting matrix’s eigenvalues? Are dynamics being studied (as it seems from the middle block in Figure 1)?

The paper is centered about “direction”, “magnitude” and ‘control”. It is unclear to me from either the introduction or the other sections on what exactly these word mean in the context of this paper. Furthermore, there are statements like “If parameter averaging supplies no preferred spectral direction, a reliable intervention must act on representation objects themselves.” What intervention?

Similar comments for the theory sections. It is completely unclear what is being proved and why these are being derived.

Questions for authors

This paper needs a significant rewrite to present: a self-contained discussion of the problem being solved, the main results and intuition, the theorems (after being motivated) need a sketch of proof to show what is novel and useful, and finally the takeaway message.

Reviewer_a18zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how model averaging in federated learning affects the diversity of learned representations, measured using von Neumann entropy (VNE). The authors show that averaging model parameters can either increase or decrease VNE and provide theoretical bounds on the resulting spectral changes. They also propose FedVNA, which encourages higher within-class diversity and alignment between client representations.

Critical strengths

Studying how model averaging affects the feature spectrum in federated learning is an interesting and important direction. The paper provides useful insights into how aggregation can change representation diversity, an aspect that is often overlooked in federated learning.

Critical weaknesses
  1. Limited practical improvement: Although FedVNA achieves noticeably higher VNE than FedAvg, this does not translate into meaningful improvements in classification accuracy. The results raise questions about the practical value of the proposed method, particularly given the additional computational and communication overhead. It would be more convincing to show benefits in other aspects, such as generalization, robustness, or performance under distribution shifts.

  2. Unclear motivation for spectral diversity and alignment: The paper does not clearly explain why increasing within-class VNE or encouraging stronger alignment between clients is necessarily beneficial. Higher within-class diversity could conflict with learning compact and discriminative representations, while forcing clients to align their feature distributions may suppress useful client-specific information, especially under heterogeneous data distributions. These potential trade-offs are not sufficiently discussed or investigated.

  3. Gap between theoretical analysis and the proposed method: The main theoretical contraction result is derived for an idealized interpolation of density matrices, while FedVNA uses a minibatch regularization objective during training. These are not equivalent, and the paper does not establish that the actual training procedure satisfies the same contraction guarantee. This makes the connection between the theoretical analysis and the proposed algorithm somewhat weak.

Questions for authors

Please see Waeknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.