- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies how parameter averaging affects class-conditional representation entropy. It provides linear examples with opposite entropy changes, bounds one-round deviation from a same-start centralized reference, and establishes disagreement contraction along an ideal density-matrix interpolation. It then proposes FedVNA, combining a shared class-conditional reference with an entropy regularizer.
Critical strengths
-
The analysis distinguishes the covariance of an averaged encoder from averaged client covariances, with explicit examples showing that aggregation can either increase or decrease VNE.
-
The experiments show spectral changes alongside largely unchanged accuracy, and report seed variation and the added computational and communication costs.
Critical weaknesses
-
The covariance decomposition, entropy counterexamples, and interpolation contraction rely largely on standard identities and convexity properties. The additional insight beyond existing VNE regularization and federated representation decorrelation remains limited.
-
Figure 4 states that zero-alignment settings use 64-dimensional features, whereas nonzero-alignment settings use 512 dimensions. The sweep therefore does not isolate the alignment term at a fixed feature dimension.
-
The experiments do not directly measure aggregation-induced entropy changes on fixed diagnostic samples or deviations from the same-start centralized reference. The practical informativeness of the theoretical analysis is therefore unclear.
-
FedVNA increases round time from about 25 seconds to 91–105 seconds and adds 10 MiB of communication per client per round in each direction, while accuracy remains close to FedAvg. Evidence is limited to CIFAR-10 with ResNet-18.
-
Clients with insufficient samples of a class contribute a maximally mixed matrix to its reference. Under strong label skew, this replaces missing information with isotropic covariance, but its effect on the reference and disagreement metric is not evaluated.
-
The training surrogate uses a ridge-regularized Gram matrix, whereas the theoretical diagnostic uses an unregularized normalized covariance. The regularized entropy can be maximal even for zero centered features, and the experiments do not assess sensitivity to ridge strength or feature scale.
Questions for authors
-
What does the class-conditional analysis establish beyond existing VNE regularization and federated representation decorrelation?
-
Why do the zero-alignment settings use 64-dimensional features while the nonzero-alignment settings use 512 dimensions? Are there existing comparisons at the same dimension that isolate the alignment term?
-
Do any existing results measure local-model and aggregated-model VNE on the same fixed diagnostic samples, or compare the aggregated model with the same-start centralized reference?
-
Given the reported computational and communication overhead, what benefit does FedVNA provide beyond increasing the entropy metric that it directly regularizes?
-
Under each heterogeneity level, what fraction of the class-reference weight comes from maximally mixed fallback matrices? How sensitive are the conclusions to this treatment of clients with insufficient class samples?
-
What weights are used to aggregate the client–class diagnostic entropies, and how are pairs with insufficient samples or zero covariance handled?
-
Are uploaded class covariances recomputed using the final fixed local model, or accumulated from features produced by changing models during the final local epoch?