acceptodds
Under review as a conference paper at ICLR 2027

MechLeash-FL: Mechanistic Leashing of Latent Features for Federated Learning under Feature Skew

Abstract

Federated Averaging (FedAvg) aggregates client models by averaging their parameters directly, so its performance implicitly depends on clients representing matching features along the same directions in the model's hidden space. We analyze what happens to this match under feature skew and representational superposition. This analysis reveals two ways that parameter averaging can degrade features: reducing feature strength and causing feature interference between clients. To mitigate these issues, we introduce MechLeash-FL, a two-phase federated training procedure that leashes clients' latent feature directions to a shared mechanistic reference during local training. MechLeash-FL uses a sparse autoencoder (SAE), shared across clients, to penalize changes in the clients' latent representations. After each aggregation, the SAE is refit to the merged model's activations and shared with the clients, providing a common reference for latent feature directions without changing the deployed model's architecture. We evaluate MechLeash-FL in two settings: a controlled toy model of superposition, where we directly measure feature shrinkage and interference, and real-data reconstruction on Office-Caltech10, a naturally feature-skewed benchmark. In a synthetic setting, MechLeash-FL reduces endpoint reconstruction error by , preserves more feature strength after aggregation, and reduces interference between clients. On Office-Caltech10, adding MechLeash-FL to FedAvg, FedProx, MOON, and Ditto reduces validation mean squared error in uncorrupted reconstruction by -.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
federated learning, feature skew, superposition, sparse autoencoders, mechanistic interpretability, model aggregation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.80 over 5 reviews, above 85% of the others.
Reviewer_BHdxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates federated learning (FL) under feature skew and superposition. Specifically, it first provides theoretical analyses highlighting the limitations of FedAvg in preserving feature representations and preventing cross-client interference under these settings. Based on these observations, the paper proposes MechLeash-FL, which regularizes model update directions using a sparse autoencoder (SAE) to capture feature representations. By alternating between updating the task models and the SAE, MechLeash-FL achieves improved reconstruction performance on both synthetic and real-world datasets.

Critical strengths
  • The paper provides theoretical analyses that motivate the design of MechLeash-FL, making the proposed approach more theoretically grounded.
Critical weaknesses
  • The evaluation is limited to synthetic datasets and the relatively small-scale Office-Caltech10 dataset, focusing exclusively on reconstruction tasks. It remains unclear whether the reported improvements can generalize to larger-scale and more realistic datasets, as well as other downstream tasks, such as classification, object detection, and semantic segmentation, which are commonly studied in federated learning.

  • The idea of leveraging latent feature representations to regularize local training in federated learning has been explored in previous studies [1][2]. However, the paper provides insufficient comparisons and discussion with these existing approaches, making it difficult to assess the novelty and advantages of the proposed method.

[1] 2024 ICLR FedImpro: Measuring and Improving Client Update in Federated Learning

[2] 2024 NeurIPS Classifier Clustering and Feature Alignment for Federated Learning under Distributed Concept Drift

Questions for authors

Besides the weakness shown in the above section, please also see the following questions:

Q1: Since the SAE is trained to capture the distributions of local feature representations and is shared across clients, could this potentially introduce privacy risks or lead to information leakage, thereby compromising data privacy in federated learning?

Q2: The two-phase training procedure in MechLeash-FL may introduce additional computational overhead and increase training time. How significant are these additional costs relative to the performance improvements achieved? It would be valuable to report and discuss the trade-off between computational efficiency and performance gains.

3 comments
AuthorsOpenReview ↗

New experiments, FedImpro/FedCCFA, privacy and cost

Thank you for the careful review and the two references. Below, we answer each point briefly. A second message, posted as a reply to this one, gives the details, tables and statistics under the same numbers. We will upload the revised paper by the 13th (AoE). The (➔...) marks show where each change goes.

1. Larger and more realistic datasets

"The evaluation is limited to synthetic datasets and the relatively small-scale Office-Caltech10 dataset" (W1) "It remains unclear whether the reported improvements can generalize to larger-scale and more realistic datasets" (W1)

We agree with this point and therefore ran DomainNet reconstruction, Office-Home classification (15.5k images, point 2) and runs with up to 32 clients on Office-Caltech10. The gains hold on those datasets, except for Ditto's personalized accuracy on Office-Home. On Office-Caltech10, they become smaller with more clients but stay significant. (➔Appendix)

2. Classification

"focusing exclusively on reconstruction tasks" (W1) "as well as other downstream tasks, such as classification" (W1)

We ran classification on Office-Home (65 classes, four domains as clients). MechLeash-FL raises the accuracy of FedAvg, FedProx and MOON by 1.9, 5.5 and 8.1 percentage points and leaves Ditto's personalized accuracy unchanged. (➔Appendix)

3. Detection and segmentation

"object detection, and semantic segmentation, which are commonly studied in federated learning." (W1)

We ran both. On prostate MRI segmentation with six hospitals as clients, MechLeash-FL raises the Dice score of FedAvg from 62.9 to 67.2 %. On object detection, we saw no benefit with the architectures we tried: the models were either too weak for the task or showed no sign of superposition, and in tasks without a sign of superposition, MechLeash-FL shows no benefit. We will add this to the limitations. (➔Appendix, Section 8)

4. FedImpro and FedCCFA

"The idea of leveraging latent feature representations to regularize local training in federated learning has been explored in previous studies [1][2]. However, the paper provides insufficient comparisons and discussion with these existing approaches, making it difficult to assess the novelty and advantages of the proposed method." (W2)

Thank you for these two references, which we had missed. We will discuss both in the related work. Like MechLeash-FL, both use latent features to guide local training. They do so with class-level statistics of the features, and MechLeash-FL with the directions of individual features. (➔Section 2)

5. Privacy

"Since the SAE is trained to capture the distributions of local feature representations and is shared across clients, could this potentially introduce privacy risks or lead to information leakage, thereby compromising data privacy in federated learning?" (Q1)

This is an important question we will discuss in the revision of the paper. The SAE adds a risk of the same kind as the model parameters FedAvg already shares: only its parameters are shared, and the same protections apply. (➔Section 5.3)

6. Cost versus gain

"The two-phase training procedure in MechLeash-FL may introduce additional computational overhead and increase training time. How significant are these additional costs relative to the performance improvements achieved? It would be valuable to report and discuss the trade-off between computational efficiency and performance gains." (Q2)

We will report it in the revised paper: depending on the baseline, 13 to 31 % more training time and 15 to 33 % lower validation MSE. Two cheaper variants keep most of the gain. (➔Appendix)

If you have further questions, we are happy to answer them.

Details for the answers above, under the same numbers. p-values are exact one-sided paired Wilcoxon tests over 10 seeds. "x/10" counts the seeds in which MechLeash-FL is better.

1. Larger datasets

Second dataset: On DomainNet (10-class subset, six domains as six clients, the paper's model and coefficients, 8 rounds), MechLeash-FL lowers the MSE by 22.9 % (FedAvg), 17.5 % (FedProx), 7.5 % (MOON) and 20.4 % (Ditto, personalized), all significant after Holm correction (all p < 0.05; MOON 8/10 seeds, the others 10/10).

More clients: we split each Office-Caltech10 domain into random parts. FedAvg + MechLeash-FL versus FedAvg:

Clients481632
lambda_rec0.30.10.030.1
MSE change-30.4 %-27.6 %-10.5 %-12.5 %
Seeds better, p10/10, 0.00110/10, 0.0019/10, 0.04210/10, 0.001

The 4-client column repeats the paper's setting on new seeds. For 8 to 32 clients, lambda_rec was chosen on three separate seeds; the table shows 10 new seeds. All gains hold after Holm correction.

2. Classification

Office-Home (65 classes, 15.5k images, four domains as clients) with the paper's encoder, a 64-dimensional bottleneck, and a linear layer on top. Validation accuracy in %:

FedAvgFedProxMOONDitto (personalized)
lambda_rec0.3110.3
Plain19.018.016.625.4
+ MechLeash-FL20.923.424.725.5
Seeds better, p7/10, 0.01910/10, 0.00110/10, 0.0014/10, 0.46

The gains for FedAvg, FedProx and MOON are significant after Holm correction. In Ditto, the leash acts only on the global model, and the personalized accuracy does not change significantly. Accuracies are around 20 % because the paper's small encoder is trained from scratch on 65 classes (chance is 1.5 %). With a ResNet-18 on DomainNet-126, the method has no effect (accuracy within 0.3 points of FedAvg). We think this network has enough dimensions to store its features without superposition. So the method helps when the model has to store more features than it has dimensions, and we will add this to the limitations.

3. Detection and segmentation

Segmentation: prostate MRI, six hospitals as clients, the paper's encoder-decoder with per-pixel class outputs. The Dice score (overlap of predicted and true prostate area, 100 % is perfect) rises from 62.9 % (FedAvg) to 67.2 % (10/10, p = 0.001), and at every hospital.

Detection: the paper's encoder-decoder with a detection head was too weak (FedAvg at most 12 % mAP, mean average precision). A pretrained ResNet-18 with a 256-dimensional layer for the SAE reaches up to 84 % mAP, but the SAE uses fewer than 100 of its 512 feature directions, so we see no sign of superposition. MechLeash-FL showed no benefit with either.

4. FedImpro and FedCCFA

FedImpro trains the upper part of the model on features sampled from a class-conditional feature distribution estimated across clients. FedCCFA groups the per-class classifiers of the clients and aligns client features to class-level anchors to handle differences in P(Y|X) between clients. Both work with labels and class-level statistics. The leash of MechLeash-FL uses no labels and acts on the directions of individual features. We will discuss this in the revised paper.

5. Privacy

Each client trains the SAE locally, and the server only averages its parameters, as for the task model. No activations or images leave a client. The SAE parameters depend on local data as the encoder weights do, so the risk is of the same kind as for the model parameters FedAvg already shares. Protections for model updates, such as secure aggregation and differential privacy on the client side, can be applied to the SAE update in the same way.

6. Cost

Training time per run on the paper's Office-Caltech10 runs (one A100 GPU) and MSE reduction:

FedAvgFedProxMOONDitto
Time without → with, min15.1 → 18.618.8 → 22.415.9 → 20.734.0 → 38.3
Extra time+23 %+19 %+31 %+13 %
Validation MSE reduction29 %33 %15 %31 %

Communication grows by a factor of 1.55 per round. With the same total upload (30 rounds with MechLeash-FL, 48 without, Figure 6), the validation MSE is still 12.7 to 33.6 % lower. Two ways to reduce the cost:

SAE budget (FedAvg)Upload per round vs. FedAvgValidation MSE reductionSeeds better, p
20 SAE epochs per round (paper setting)1.55x30.4 %10/10, 0.001
10 / 5 / 2 / 1 epochs1.55x29.8 / 29.4 / 28.2 / 23.0 %10/10, 0.001 each
Refit every 4 / 12 rounds1.14x / 1.05x23.0 / 15.2 %10/10, 0.001 / 7/10, 0.014
Fitted once (SAE-Frozen)about 1x10.4 %7/10, 0.042

Refitting only every r rounds skips the fit in r-1 of r rounds, and clients keep the last averaged SAE in between. The first two rows use new seeds (paper: 29.1 %), the last two use the paper's seeds.

Dear Reviewer,

Thank you again for your review. We uploaded the revised paper on 13 Nov (AoE). It includes the changes described above. If any questions or concerns remain, please let us know and we will gladly address them.

Best regards, The authors

Reviewer_RafJOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates feature shrinkage and cross-client interference caused by federated model aggregation under feature skew and superposition. The authors propose MechLeash-FL, which uses a shared sparse autoencoder (SAE) to regularize latent representations during local training and encourage feature consistency across clients. The method is evaluated on synthetic data and Office-Caltech10 reconstruction tasks, showing improvements over FedAvg, FedProx, MOON, and Ditto.

Critical strengths
  1. Mechanism-motivated approach. The paper provides an intuitive analysis of feature shrinkage and cross-client interference in federated aggregation, offering a meaningful motivation for the proposed SAE-based regularization.
  2. Performance improvements. MechLeash-FL achieves reconstruction improvements across four federated learning baselines. The synthetic experiments also provide supporting evidence through feature strength and interference measurements.
Critical weaknesses

1: Limited Novelty. Prior works such as FedMA (ICLR 2020), FedBN (ICLR 2021), MOON (CVPR 2021), and FedRep (ICML 2021) have already explored representation mismatch, feature heterogeneity in federated learning. In particular, the experiments lack direct comparisons with representative feature-alignment methods and simpler representation regularization strategies, making it difficult to assess the unique contribution of MechLeash-FL.

2: Insufficient Experiments. Real-data experiments are limited to Office-Caltech10, with only four clients and 62 training images per client. Standard downstream classification experiments, such as CIFAR-10/CIFAR-100 with ResNet-18 under feature-skew settings, are missing. Furthermore, several important ablations are absent, including comparisons with a standard autoencoder, a fixed SAE, and simpler representation regularization methods.

Questions for authors
  1. How does MechLeash-FL compare with existing feature-alignment methods under comparable feature-skew settings?
  2. Are the performance gains primarily due to feature alignment or general regularization? How does the method perform when the shared SAE is replaced with a standard AE or a fixed SAE?
  3. How does MechLeash-FL perform under different levels of feature skew? Do the improvements remain consistent?
5 comments
AuthorsOpenReview ↗

Related methods, ablations, classification and skew levels

Thank you for the pointers to related methods and the list of ablations. Below, we answer each point briefly. A second message, posted as a reply to this one, gives the details, tables and statistics under the same numbers. We will upload the revised paper by the 13th (AoE). The (➔...) marks show where each change goes.

1. Related methods

"Prior works such as FedMA (ICLR 2020), FedBN (ICLR 2021), MOON (CVPR 2021), and FedRep (ICML 2021) have already explored representation mismatch, feature heterogeneity in federated learning." (W1)

These works address the same problem of mismatched representations, at the level of neurons (FedMA), normalization statistics (FedBN), the whole representation (MOON) and the output layer (FedRep). We will discuss them in more detail. MechLeash-FL looks at a finer level: the directions of individual features stored in superposition. (➔Section 2)

2. Comparison with feature-alignment methods

"In particular, the experiments lack direct comparisons with representative feature-alignment methods" (W1) "How does MechLeash-FL compare with existing feature-alignment methods under comparable feature-skew settings?" (Q1)

MOON, which pulls each client's representation toward the global model's, is our feature-alignment baseline in the same setting. FedAvg + MechLeash-FL is on par with MOON on Office-Caltech10 and better on our two new datasets, DomainNet and Office-Home (points 4 and 6), and MechLeash-FL can also be added to MOON and improves it. (➔Section 2, Appendix)

3. Ablations: alignment or general regularization?

"and simpler representation regularization strategies, making it difficult to assess the unique contribution of MechLeash-FL." (W1) "Furthermore, several important ablations are absent, including comparisons with a standard autoencoder, a fixed SAE, and simpler representation regularization methods." (W2) "Are the performance gains primarily due to feature alignment or general regularization? How does the method perform when the shared SAE is replaced with a standard AE or a fixed SAE?" (Q2)

We agree these ablations were missing and ran all three: a standard autoencoder, a fixed SAE, and an L2 anchor to the server model's activations. All three use the same lambda_rec as MechLeash-FL. In the toy model, MechLeash-FL beats all three and the standard autoencoder gives no gain. On Office-Caltech10, MechLeash-FL beats the fixed SAE and the anchor and is on par with the standard autoencoder, so there we cannot separate alignment from general regularization (see the second message). (➔Appendix)

4. Only one real dataset

"Real-data experiments are limited to Office-Caltech10" (W2)

We ran DomainNet reconstruction and Office-Home classification. The gains hold on both, except for Ditto's personalized accuracy on Office-Home. (➔Appendix)

5. Few clients and few images

"with only four clients and 62 training images per client." (W2)

We ran Office-Caltech10 with up to 32 clients: with lambda_rec chosen per client count, MechLeash-FL added to FedAvg lowers the validation MSE by 27.6 % with 8, 10.5 % with 16 and 12.5 % with 32 clients. In the Office-Home classification, each client has 1,900 to 3,600 training images. (➔Appendix)

6. Classification on CIFAR with ResNet-18

"Standard downstream classification experiments, such as CIFAR-10/CIFAR-100 with ResNet-18 under feature-skew settings, are missing." (W2)

We ran classification on Office-Home because its clients differ by image domain, which gives natural feature skew. With a ResNet-18 on DomainNet-126, the method has no effect, which we will explain and add to the limitations. (➔Appendix, Section 8)

7. Levels of feature skew

"How does MechLeash-FL perform under different levels of feature skew? Do the improvements remain consistent?" (Q3)

Across the additional skew levels we tested, the reduction of validation MSE stays at 27 to 31 %. (➔Appendix)

We are happy to discuss any further questions.

Details for the answers above, under the same numbers. p-values are exact one-sided paired Wilcoxon tests over 10 seeds. "x/10" counts the seeds in which MechLeash-FL is better.

1. Related methods

FedMA matches neurons across clients by permutation before averaging. In our setting, all clients start every round from the same model, so neurons likely stay matched [1]. Under superposition, the feature directions inside a layer drift, which a permutation of neurons cannot undo. FedBN keeps BatchNorm statistics local. Our models have no normalization layers. FedRep is a personalization method that keeps the output layer local; our personalization baseline is Ditto.

2. MOON

Compared directly, FedAvg + MechLeash-FL is on par with MOON on Office-Caltech10 (MSE 0.0201 vs 0.0200) and better on DomainNet (0.0325 vs 0.0362, 9/10; point 4) and on Office-Home classification (accuracy 20.9 vs 16.6 %, 9/10; point 6). Added to MOON, it lowers the MSE by 15 % on Office-Caltech10 and raises the accuracy by 8.1 points on Office-Home (10/10 each).

3. Ablations

AE: a dense ReLU autoencoder (our SAE without top-k), refitted every round. SAE-Frozen: an SAE fitted once after the warm-up round. Anchor: an L2 penalty that pulls the hidden activations toward those of the model sent by the server, divided by the mean squared activation like our leash. MSE (lower is better):

Toy MSEOffice-Caltech10 MSE
FedAvg0.05170.0284
+ MechLeash-FL0.04340.0201
+ AE0.0517 (10/10, 0.001)0.0202 (6/10, 0.46)
+ SAE-Frozen0.0497 (10/10, 0.001)0.0254 (10/10, 0.001)
+ Anchor0.0507 (10/10, 0.001)0.0225 (9/10, 0.003)

Brackets: (seeds in which MechLeash-FL is better than that row, p). The toy column is a rerun, so it differs slightly from Section 7.1.

In the toy model, MechLeash-FL also raises the cosine between the clients' feature directions from 0.13 to 0.34 (10/10). Why the AE is on par on Office-Caltech10: with only 248 training images, the 256-dimensional hidden layer of the FedAvg model has an effective rank of about 23 (the exponential of the entropy of the eigenvalue distribution of the activation covariance, which equals d when the variance is spread equally over d directions). A dense autoencoder can then reconstruct the layer as well as a sparse one. Sparsity matters when the model has to store more features than it has dimensions, as in the toy model.

4. Second dataset

On DomainNet (10-class subset, six domains as clients, the paper's model and coefficients, 8 rounds), MechLeash-FL lowers the MSE by 22.9 % (FedAvg), 17.5 % (FedProx), 7.5 % (MOON) and 20.4 % (Ditto, personalized), all significant after Holm correction (MOON 8/10 seeds, the others 10/10).

5. More clients

We split each Office-Caltech10 domain into random parts. FedAvg + MechLeash-FL versus FedAvg:

Clients481632
lambda_rec0.30.10.030.1
MSE change-30.4 %-27.6 %-10.5 %-12.5 %
Seeds better, p10/10, 0.00110/10, 0.0019/10, 0.04210/10, 0.001

The 4-client column repeats the paper's setting on new seeds. For 8 to 32 clients, lambda_rec was chosen on three separate seeds. The table shows 10 new seeds. All gains hold after Holm correction.

6. Classification

Office-Home (65 classes, four domains as clients) with the paper's encoder, a 64-dimensional bottleneck, and a linear layer on top. Validation accuracy in %:

FedAvgFedProxMOONDitto (personalized)
lambda_rec0.3110.3
Plain19.018.016.625.4
+ MechLeash-FL20.923.424.725.5
Seeds better, p7/10, 0.01910/10, 0.00110/10, 0.0014/10, 0.46

The gains for FedAvg, FedProx and MOON are significant after Holm correction. In Ditto, the leash acts only on the global model, and the personalized accuracy does not change significantly. Accuracies are around 20 % because the paper's small encoder is trained from scratch on 65 classes (chance is 1.5 %). With a standard ResNet-18 on DomainNet-126, the method has no effect (accuracy within 0.3 points of FedAvg). We think this network has enough dimensions to store its features without superposition.

7. Skew levels

We spread the Office-Caltech10 images of all domains over the four clients with a Dirichlet distribution over domains (smaller alpha, more skew). The MSE reduction is 29.1, 27.4, 29.0, 28.8 and 30.9 % for alpha = 0.1, 0.3, 1, 10 and 100 (10/10, p = 0.001 each), compared with 29.1 % with one domain per client. We think the gain does not shrink because each client has only about 62 images on average. Even with the same mix of domains, each client sees a different small sample, so which features are frequent still differs between clients, and 100 local epochs per round let the client models drift apart.

[1] Frankle et al. Linear Mode Connectivity and the Lottery Ticket Hypothesis. ICML 2020.

Dear Reviewer,

Thank you again for your review. We uploaded the revised paper on 13 Nov (AoE). It includes the changes described above. If any questions or concerns remain, please let us know and we will gladly address them.

Best regards, The authors

Reviewer_CLHLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studied the critical issue, i.e., averaging model weights may reduce feature strength and lead to a feature interference between clients. The paper leashes clients’ latent feature direction to a shared mechanistic reference when training locally. It then uses a sparse autoencoder and shares it among clients, in order to penalize changes in the clients’ latent representations. Then, the sparse autoencoder adjusts to the merged model’s activations and share with clients to provide a common reference for latent feature direction but keep the model’s architecture.

Critical strengths

The paper analyzes the feature level performance of federated averaging under feature skew. It contains the feature shrinkage from the cross-client overlap that can turn into interference. The paper uses a shared sparse autoencoder to keep client’s latent feature direction, but does not change the deployed model’s architecture. The performance evaluation connects reconstruction behavior to representation geometry in a controlled synthetic setting. The results show some advantages compared with several baselines. The paper presents in a logical way with the start of feature-level analysis and shows the observations as the motivation to further improve.

Critical weaknesses

The novelty is not well highlighted. The method includes several existing techniques. Then, what is the novelty? The evaluation needs to be strengthened. Although the performance evaluation shows some results, it still lacks the validation of the method’s real-world generalization. It is doubtful that the proposed method may be limited with certain scenario or dataset. Expanding the real-data evaluation would address this issue.

Questions for authors

What is the rationale to set up experiments in the performance evaluation? How could the insights gained from the current setting reflect and validate the proposed method?

3 comments
AuthorsOpenReview ↗

Novelty, new real-data experiments and design rationale

We are grateful for the review and the questions. Below, we answer each point briefly. A second message, posted as a reply to this one, gives the details, tables and statistics under the same numbers. We will upload the revised paper by the 13th (AoE). The (➔...) marks show where each change goes.

1. Novelty

"The novelty is not well highlighted. The method includes several existing techniques. Then, what is the novelty?" (W)

We agree that the paper should state this more clearly. FedAvg and sparse autoencoders (SAEs) are known. New are (i) a feature-level analysis of why parameter averaging damages representations under feature skew and superposition, and (ii) the use of a shared SAE during training as a reference for feature directions. To our knowledge, this is the first use of SAEs to improve federated averaging. (➔Sections 1 and 2)

2. Real-data evaluation

"The evaluation needs to be strengthened. Although the performance evaluation shows some results, it still lacks the validation of the method’s real-world generalization. It is doubtful that the proposed method may be limited with certain scenario or dataset. Expanding the real-data evaluation would address this issue." (W)

We understand the concern that the results could depend on one dataset. We ran five new real-data experiments: more clients, a second dataset (DomainNet), weaker feature skew, classification on Office-Home, and segmentation of prostate MRI. The gains are significant in most settings (on DomainNet for all four baselines, 7.5 to 22.9 %), and we also report where they are small or not significant. (➔Appendix)

3. Rationale of the experiments

"What is the rationale to set up experiments in the performance evaluation? How could the insights gained from the current setting reflect and validate the proposed method?" (Q)

The toy model lets us measure the two problems our analysis predicts, because there the true features are known. Office-Caltech10 reconstruction is the closest real-data setting, because it makes superposition likely and its domains differ in their features. (➔Section 6)

Please let us know if anything is still unclear.

Details for the answers above, under the same numbers. p-values are exact one-sided paired Wilcoxon tests over 10 seeds. "x/10" counts the seeds in which MechLeash-FL is better.

1. Novelty

At the level of single features, features that only one client represents strongly get weaker after averaging, and features that did not conflict on either client start to interfere after merging. We measure both in the toy model. With our method, the loss of feature strength after averaging drops from 42 % to 23 %, and the merge barrier (loss after averaging minus mean loss of the client models) changes from +0.0095 to -0.0189, so averaging helps where it hurt before. SAEs are normally used after training to find the features a network stores. We use a shared SAE during training, refitted every round, which gives all clients the same set of feature directions, and a penalty on its reconstruction error keeps each client's features close to these directions (Proposition 1). Our regularization also differs from prior work: FedProx regularizes the weights, MOON the similarity of a client's representation to the global model's, FedBN the normalization statistics, and we regularize the directions of latent features. The method can therefore be combined with FedProx, MOON and Ditto and improves them on reconstruction.

2. Real-data evaluation

Besides the masked reconstruction in Appendix D.5:

More clients: we split each Office-Caltech10 domain into random parts. FedAvg + MechLeash-FL versus FedAvg:

Clients481632
lambda_rec0.30.10.030.1
MSE change-30.4 %-27.6 %-10.5 %-12.5 %
Seeds better, p10/10, 0.00110/10, 0.0019/10, 0.04210/10, 0.001

The 4-client column repeats the paper's setting on new seeds. For 8 to 32 clients, lambda_rec was chosen on three separate seeds; the table shows 10 new seeds. All gains hold after Holm correction. With 32 clients, each client has 7 to 8 images.

Second dataset: On DomainNet (10-class subset, six domains as six clients, the paper's model and coefficients, 8 rounds), MechLeash-FL lowers the MSE by 22.9 % for FedAvg, 17.5 % for FedProx, 7.5 % for MOON and 20.4 % for Ditto's personalized models (MOON 8/10, p = 0.014; the others 10/10, p = 0.001; all significant after Holm correction).

Weaker feature skew: We spread the Office-Caltech10 images of all domains over the four clients with a Dirichlet distribution over domains (smaller alpha, more skew). The MSE reduction is 29.1, 27.4, 29.0, 28.8 and 30.9 % for alpha = 0.1, 0.3, 1, 10 and 100 (10/10, p = 0.001 each), compared with 29.1 % with one domain per client. We think the gain does not shrink because each client has only about 62 images on average. Even with the same mix of domains, each client sees a different small sample, so which features are frequent still differs between clients, and 100 local epochs per round let the client models drift apart.

Classification: Office-Home (65 classes, 15.5k images, four domains as clients) with the paper's encoder, a 64-dimensional bottleneck, and a linear layer on top. Validation accuracy in %:

FedAvgFedProxMOONDitto (personalized)
lambda_rec0.3110.3
Plain19.018.016.625.4
+ MechLeash-FL20.923.424.725.5
Seeds better, p7/10, 0.01910/10, 0.00110/10, 0.0014/10, 0.46

The gains for FedAvg, FedProx and MOON are significant after Holm correction. In Ditto, the leash acts only on the global model (trained like FedAvg), and the personalized accuracy shows no significant difference. Accuracies are around 20 % because the paper's small encoder is trained from scratch on 65 classes (chance is 1.5 %). We chose the bottleneck width 64 in a test over 32, 64 and 128. At width 128, the gain is similar (accuracy 20.0 to 22.0 %, 3 seeds).

Segmentation: prostate MRI with six hospitals as clients and the paper's encoder-decoder with per-pixel class outputs. With lambda_rec = 0.3, the Dice score (overlap of predicted and true prostate area, 100 % is perfect) rises from 62.9 % for FedAvg to 67.2 % for FedAvg + MechLeash-FL (10/10, p = 0.001), and at every hospital.

3. Rationale

In the toy model, we can check whether the method removes the two predicted problems, weaker features and interference. In Office-Caltech10 reconstruction, perfect reconstruction needs more features than the bottleneck has dimensions, and the four domains as clients differ in their image features but not in their labels. Classification can need fewer features than reconstruction, so the leash may have less to act on. We therefore chose Office-Home with 65 classes for classification.

Dear Reviewer,

Thank you again for your review. We uploaded the revised paper on 13 Nov (AoE). It includes the changes described above. If any questions or concerns remain, please let us know and we will gladly address them.

Best regards, The authors

Reviewer_5JTSOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper analyzes FedAvg under feature skew and representational superposition. Its starting observation is that parameter averaging implicitly assumes clients represent the same features along the same hidden directions, and that under feature skew this assumption breaks in two ways: features represented strongly by only one client get shrunk by averaging, and features never in conflict on either client alone can begin to interfere after merging. The proposed MechLeash-FL shares a TopK sparse autoencoder (SAE) across clients and alternates two phases per round: clients train their task model with an SAE reconstruction penalty while the SAE is frozen, then fit local SAE copies to their activations while the task model is frozen, with the server aggregating both; the SAE is discarded after training, leaving the deployed architecture unchanged. The paper formalizes the connection between a directional leash on features and the SAE reconstruction loss. Evaluation covers a controlled toy model of superposition, directly measuring feature strength and pairwise interference via the Gram matrix of learned directions, and image reconstruction on Office-Caltech10, where the four domains form naturally feature-skewed clients. MechLeash-FL reduces endpoint reconstruction error by 14.82% in the synthetic setting, reduces aggregation-induced feature shrinkage from 42.1% to 23.1%, flips the merge barrier from harmful (+0.0095) to helpful (-0.0189), and lowers validation MSE by 15.29%-33.08% when added to FedAvg, FedProx, MOON, and Ditto, with exact paired Wilcoxon tests and Holm correction, at about 1.54x communication.

Critical strengths
  • A genuinely new connection between mechanistic interpretability and federated optimization, with mechanism-level evidence. To tell the truth, I am quite impressed by how the paper closes the loop between analysis and method: the toy-model study directly measures the two predicted failure modes — shrinkage (42.1% vs. 23.1%) and interference through the Gram matrix — and the sign flip of the merge barrier is a memorable result that few FL papers can offer at this level of mechanistic detail. This is the primary driver of my positive assessment.
  • Rigor and honesty. The statistics (exact paired Wilcoxon, Holm correction across baselines) are more careful than most FL papers I review; the matched-communication-budget comparison in the appendix addresses the obvious overhead objection; and the limitations, AI-use, and reproducibility statements are exemplary. The consistent gains across four algorithmically distinct baselines (parameter regularization, representation regularization, personalization) suggest the leash complements rather than competes with existing approaches.
Critical weaknesses
  • The evaluation is limited to reconstruction tasks and very small federations. The motivating story of feature skew is arguably most consequential for classification under label/feature skew, where most FL heterogeneity benchmarks live, yet both testbeds are reconstructive, with two clients in the synthetic study and four in the real-data one. Whether the leash helps or hurts when the objective is discriminative, and how SAE aggregation behaves at moderate client counts, remain open, and this bounds the paper's current impact. The authors acknowledge this, which I appreciate, but it is still the main factor capping my rating.
  • Overhead accounting is incomplete on the compute side. Communication is reported (1.54x, with a matched-budget comparison), but fitting a TopK SAE per client per round is nontrivial, and in larger models SAE training is notoriously expensive and unstable (dead features, reconstruction plateaus). Without wall-clock/FLOP overhead and an analysis of how the method degrades with an undertrained SAE, the practicality claim rests on the toy scale.
Questions for authors
  • [Clarification] Does the leash transfer to a discriminative objective? Even one classification experiment under feature skew would materially change my assessment of the method's scope.
  • [Clarification] How do results change with more clients (e.g., 20-50 via synthetic domain splits), given that FL heterogeneity pathologies interact with client count?
  • [Clarification] What are the wall-clock/FLOP overheads of the per-round SAE fitting, and how sensitive is the method to an undertrained SAE (e.g., fewer SAE steps, or refitting every r rounds)?
  • [Clarification] The leash anchors clients to the previous round's merged SAE geometry; is there a measurable tension with personalization, e.g., per-client vs. global performance under Ditto with the leash?
  • [Suggestion] Practitioner guidance on setting lambda_rec (the paper notes it is the sensitive hyperparameter) would improve usability.
  • [Minor] The identical p=0.00391 across the four real-data comparisons is plausible with ten repetitions (2^-8) but worth stating explicitly; please define "merge barrier" with a formula rather than only in text.
3 comments
AuthorsOpenReview ↗

Classification, more clients, SAE overhead and Ditto

Thank you for the detailed review and the concrete suggestions. Below, we answer each point briefly. A second message, posted as a reply to this one, gives the details, tables and statistics under the same numbers. We will upload the revised paper by the 13th (AoE). The (➔...) marks show where each change goes.

1. Classification

"The evaluation is limited to reconstruction tasks" (W1) "Whether the leash helps or hurts when the objective is discriminative" (W1) "Does the leash transfer to a discriminative objective? Even one classification experiment under feature skew would materially change my assessment of the method's scope." (Q1)

To close this gap, we ran classification on Office-Home (65 classes, four domains as clients). The leash helps: it raises the accuracy of FedAvg, FedProx and MOON by 1.9, 5.5 and 8.1 percentage points and leaves Ditto's personalized accuracy unchanged. (➔Appendix)

2. More clients and SAE aggregation

"and how SAE aggregation behaves at moderate client counts, remain open" (W1) "How do results change with more clients (e.g., 20-50 via synthetic domain splits), given that FL heterogeneity pathologies interact with client count?" (Q2)

We split each domain into random parts, up to 32 clients. The averaged SAE reconstructs the merged model's activations with an error below 0.5 % for all client counts. With lambda_rec chosen per client count, MechLeash-FL lowers the validation MSE by 27.6 % with 8 clients, 10.5 % with 16 and 12.5 % with 32 clients. On Office-Home classification, the gain is not significant with 24 and 32 clients. We will add this to the limitations. (➔Appendix, Section 8)

3. Compute overhead

"Overhead accounting is incomplete on the compute side." (W2) "What are the wall-clock/FLOP overheads of the per-round SAE fitting" (Q3)

We will report wall-clock time in the revised paper. On the paper's runs, the SAE adds 13 to 31 % training time, depending on the baseline. (➔Appendix)

4. Stability of SAE training

"in larger models SAE training is notoriously expensive and unstable (dead features, reconstruction plateaus)" (W2)

In our runs, the SAE fit did not get stuck: its reconstruction error falls from 2.2 % to about 0.1 % within 12 rounds. (➔Appendix)

5. Undertrained SAE

"Without wall-clock/FLOP overhead and an analysis of how the method degrades with an undertrained SAE, the practicality claim rests on the toy scale." (W2) "and how sensitive is the method to an undertrained SAE (e.g., fewer SAE steps, or refitting every r rounds)?" (Q3)

We ran both variants you suggest. The method degrades gracefully: most of the gain remains with fewer SAE epochs, and refitting only every few rounds keeps a clear gain at lower cost. (➔Appendix)

6. Personalization

"The leash anchors clients to the previous round's merged SAE geometry; is there a measurable tension with personalization, e.g., per-client vs. global performance under Ditto with the leash?" (Q4)

We see no measurable tension. On Office-Caltech10, both Ditto's global model and every client's personalized model improve, and on Office-Home the personalized accuracy is unchanged. (➔Appendix)

7. Setting lambda_rec

"Practitioner guidance on setting lambda_rec (the paper notes it is the sensitive hyperparameter) would improve usability." (S)

Based on our experiments, we suggest 0.3 as default, smaller values (0.03 to 0.1) with many clients, and the grid {0.03, 0.1, 0.3, 1} if tuning is possible. (➔Appendix)

8. Identical p-values

"The identical p=0.00391 across the four real-data comparisons is plausible with ten repetitions (2^-8) but worth stating explicitly" (M)

Yes, this is the reason. Appendix D.2 explains it, and Section 7.2 will link to it. (➔Section 7.2)

9. Merge barrier formula

"please define "merge barrier" with a formula rather than only in text." (M)

We will replace equation 16 with a definition for any number of clients C. (➔Section 6.1)

Thank you again, and please let us know if we can clarify anything else.

Details for the answers above, under the same numbers. p-values are exact one-sided paired Wilcoxon tests over 10 seeds. "x/10" counts the seeds in which MechLeash-FL is better.

1. Classification

Reconstruction was our main task because it is close to the toy model: perfect reconstruction needs more features than the bottleneck has dimensions, which makes superposition likely. For classification, we chose Office-Home because 65 classes need more features than a task with few classes. We use the paper's encoder with a 64-dimensional bottleneck and a linear layer on top. Validation accuracy in %:

FedAvgFedProxMOONDitto (personalized)
lambda_rec0.3110.3
Plain19.018.016.625.4
+ MechLeash-FL20.923.424.725.5
Seeds better, p7/10, 0.01910/10, 0.00110/10, 0.0014/10, 0.46

The gains for FedAvg, FedProx and MOON are significant after Holm correction. In Ditto, the leash acts only on the global model (trained like FedAvg), and the personalized accuracy shows no significant difference. Accuracies are around 20 % because the paper's small encoder is trained from scratch on 65 classes (chance is 1.5 %). We chose the bottleneck width 64 in a test over 32, 64 and 128. At width 128, the gain is similar (accuracy 20.0 to 22.0 %, 3 seeds).

2. More clients

We split each Office-Caltech10 domain into random parts. FedAvg + MechLeash-FL versus FedAvg:

Clients481632
lambda_rec0.30.10.030.1
MSE change-30.4 %-27.6 %-10.5 %-12.5 %
Seeds better, p10/10, 0.00110/10, 0.0019/10, 0.04210/10, 0.001

The 4-client column repeats the paper's setting on new seeds. For 8 to 32 clients, lambda_rec was chosen on three separate seeds; the table shows 10 new seeds. All gains hold after Holm correction. The relative reconstruction error of the averaged SAE (its squared error on the merged model's validation activations, divided by their squared norm, mean of the last five rounds) is 0.06, 0.19, 0.44 and 0.17 % with 4, 8, 16 and 32 clients. On Office-Home (FedAvg), the accuracy gain is 1.6 percentage points at 8 and 16 clients (10/10 each) and not significant at 24 and 32 clients (0.4 and 0.5 points).

3. Compute overhead

Training time per run on the paper's Office-Caltech10 runs (one A100 GPU): FedAvg 15.1 → 18.6 min (+23 %), FedProx 18.8 → 22.4 (+19 %), MOON 15.9 → 20.7 (+31 %), Ditto 34.0 → 38.3 (+13 %), without → with MechLeash-FL. With the same total upload, MechLeash-FL still has a 12.7 to 33.6 % lower validation MSE (Figure 6).

4. Stability

The relative reconstruction error of the averaged SAE is 2.2 % after the first leashed round and 0.09 % on average over rounds 12 to 48 (FedAvg, Office-Caltech10).

5. Undertrained SAE

SAE budget (FedAvg)Upload per round vs. FedAvgValidation MSE reductionSeeds better, p
Paper setting: 20 SAE epochs, refit every round1.55x30.4 %10/10, 0.001
10 / 5 / 2 / 1 SAE epochs1.55x29.8 / 29.4 / 28.2 / 23.0 %10/10, 0.001 each
Refit every 4 / 12 rounds1.14x / 1.05x23.0 / 15.2 %10/10, 0.001 / 7/10, 0.014
Fitted once (SAE-Frozen)about 1x10.4 %7/10, 0.042

Refitting only every r rounds skips the fit in r-1 of r rounds, and clients keep the last averaged SAE in between. The first two rows use new seeds (paper: 29.1 %), the last two the paper's seeds.

6. Personalization

Office-Caltech10 validation MSE, Ditto versus Ditto + MechLeash-FL: global model 0.0284 to 0.0201, personalized models 0.0282 to 0.0195 (10/10, p = 0.001 each). Per client (personalized): Amazon 0.0312 to 0.0230, Caltech 0.0441 to 0.0302, DSLR 0.0189 to 0.0120, Webcam 0.0185 to 0.0128. On Office-Home, the personalized accuracy is 25.4 % without and 25.5 % with the leash (4/10, p = 0.46).

7. lambda_rec

All our results use lambda_rec between 0.01 and 1, with the leash divided by the mean squared activation (toy model 0.01, Office-Caltech10 0.03 to 1, Office-Home 0.3 to 1). Larger values were too restrictive. With more clients, smaller values worked better (0.1, 0.03 and 0.1 for 8, 16 and 32 clients).

8. p-values

With all 10 differences in the same direction, the exact one-sided p-value is 1/1024, and Holm correction over four tests gives 4/1024 = 0.0039 for all four.

9. Merge barrier

Br=Lpool(θr+1)−1C∑cLpool(θr,c)B^r = L_{pool}(\theta^{r+1}) - \frac{1}{C}\sum_c L_{pool}(\theta^{r,c}), where θr,c\theta^{r,c} is client c's model after round r, θr+1\theta^{r+1} is their average, and LpoolL_{pool} is the mean validation loss over all clients. B > 0 means that averaging hurts.

Edit (18 Nov): communication factor updated to 1.55x to match the revised paper (exact value 1.548, previously rounded down).

Dear Reviewer,

Thank you again for your review. We uploaded the revised paper on 13 Nov (AoE). It includes the changes described above. If any questions or concerns remain, please let us know and we will gladly address them.

Best regards, The authors

Reviewer_s8HXOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how feature skew affects model aggregation in federated learning. The authors show that averaging local models can weaken learned features and introduce interference between features. They propose MechLeash-FL, which uses a shared sparse autoencoder (SAE) to constrain local representations. The method alternates between training the task model and updating the SAE. Experiments on synthetic data show less feature shrinkage and interference. On Office-Caltech10, the method improves reconstruction performance across four FL baselines.

Critical strengths
  1. The paper provides an interesting explanation of why parameter averaging can hurt model performance under feature skew. The analysis of feature shrinkage and interference is useful for understanding this problem.

  2. The proposed method is interesting and can be combined with existing FL methods. The results are consistent across four baselines. I also appreciate that the synthetic experiments examine feature geometry instead of only reporting reconstruction errors.

Critical weaknesses
  1. I am not fully convinced that the SAE reconstruction loss actually aligns features across clients. The theoretical result relies on a simplified TopK-1 setting, while the experiments use larger values of k. A low reconstruction error does not necessarily mean that the same features are aligned. The improvement might also come from general representation regularization. Comparisons with standard autoencoders or other regularization methods are missing.

  2. The experimental evaluation is limited. The paper only considers reconstruction tasks, with two clients in the synthetic experiments and four clients on Office-Caltech10. It is unclear whether the method works well for classification or larger federations. Also, training an SAE in every round introduces additional computation, but the paper does not report the training overhead.

Questions for authors
  1. The equivalence between directional leashing and SAE reconstruction is proved for TopK-1. Does this result still hold when k is larger? How can the authors verify that the SAE aligns the same features across clients?

  2. Could the improvement simply come from the additional reconstruction regularization? Have the authors compared the method with a standard autoencoder or a simpler representation regularizer?

  3. How much additional training time does the SAE introduce? Also, what happens if the SAE is not fully trained in each round?

5 comments
AuthorsOpenReview ↗

Alignment, ablations, classification, clients and SAE cost

Thank you for the careful reading and the precise questions. Below, we answer each point briefly. A second message, posted as a reply to this one, gives the details, tables and statistics under the same numbers. We will upload the revised paper by the 13th (AoE). The (➔...) marks show where each change goes.

1. Larger k

"The theoretical result relies on a simplified TopK-1 setting, while the experiments use larger values of k." (W1) "The equivalence between directional leashing and SAE reconstruction is proved for TopK-1. Does this result still hold when k is larger?" (Q1)

We agree that a TopK-1 result alone does not cover the setting of our experiments. We extended Proposition 1 to q active features and a TopK-q SAE. On average over the inputs, the SAE loss is still at least a positive constant times the directional leash. The overlap between active atoms adds a term that is small when few features are active at once. What does not carry over is the statement for each single input and each single feature. (➔Section 5.2, Appendix)

2. Do the features actually align?

"I am not fully convinced that the SAE reconstruction loss actually aligns features across clients." (W1) "A low reconstruction error does not necessarily mean that the same features are aligned." (W1) "How can the authors verify that the SAE aligns the same features across clients?" (Q1)

We measured alignment directly in the toy model, where the true features are known: MechLeash-FL raises the mean cosine between the two clients' directions for the same feature from 0.13 to 0.34 (1 means the same direction). So the leash partly aligns the clients' features, and averaging weakens them less (loss of feature strength 23 % instead of 42 %). On real data, the true features are unknown, so alignment cannot be measured there. (➔Section 7.1)

3. General regularization or alignment?

"The improvement might also come from general representation regularization. Comparisons with standard autoencoders or other regularization methods are missing." (W1) "Could the improvement simply come from the additional reconstruction regularization? Have the authors compared the method with a standard autoencoder or a simpler representation regularizer?" (Q2)

To close this point, we ran three ablations that change only the regularizer: a dense autoencoder, a frozen SAE, and an L2 anchor to the server model's activations. All three use the same coefficient as MechLeash-FL. In the toy model, MechLeash-FL beats all three, and the dense autoencoder gives no gain at all. On Office-Caltech10, MechLeash-FL beats the frozen SAE and the anchor and is on par with the dense autoencoder, for a reason we explain in the second message. (➔Appendix)

4. Classification

"The experimental evaluation is limited. The paper only considers reconstruction tasks" (W2) "It is unclear whether the method works well for classification" (W2)

We ran classification on Office-Home (65 classes, four domains as clients). MechLeash-FL raises the accuracy of FedAvg, FedProx and MOON by 1.9, 5.5 and 8.1 percentage points and leaves Ditto's personalized accuracy unchanged. (➔Appendix)

5. Larger federations

"with two clients in the synthetic experiments and four clients on Office-Caltech10" (W2) "or larger federations" (W2)

We split each domain into random parts, up to 32 clients. With lambda_rec chosen per client count, MechLeash-FL lowers the validation MSE by 27.6 % with 8 clients, 10.5 % with 16 and 12.5 % with 32 clients (7 to 8 images per client). On Office-Home classification, the gain is not significant with 24 and 32 clients. We will add this to the limitations. (➔Appendix, Section 8)

6. Training overhead

"Also, training an SAE in every round introduces additional computation, but the paper does not report the training overhead." (W2) "How much additional training time does the SAE introduce?" (Q3)

We will report it in the revised paper. In the paper's runs, the SAE adds 13 to 31 % training time, depending on the baseline, and the validation MSE is 15 to 33 % lower. (➔Appendix)

7. SAE not fully trained

"Also, what happens if the SAE is not fully trained in each round?" (Q3)

Most of the gain remains: with 2 instead of 20 SAE epochs per round, the reduction of validation MSE drops only from 30.4 to 28.2 %, and even 1 epoch or a refit only every 4 rounds keeps a clear gain. (➔Appendix)

Please let us know if you would like more details.

Details for the answers above, under the same numbers. p-values are exact one-sided paired Wilcoxon tests over 10 seeds. "x/10" counts the seeds in which MechLeash-FL is better.

1. Larger k

As for k = 1, the result applies to an idealized SAE built from the leash directions, with k = q. Assumptions: (i) top-k selects exactly the atoms of the active features, and (ii) (q-1)μ < 1, where μ=max⁡i≠j∣cos⁡(W^i,W^j)∣\mu = \max_{i \neq j} |\cos(\hat{W}_i, \hat{W}_j)|. Result: the expected SAE loss is at least γ times the directional leash of Proposition 1, with each feature weighted by its squared norm. γ depends only on the leash directions and the input distribution, and is positive under a mild condition on them. The overlap term moves the minimizer away from full alignment by an amount of second order in μ. Two things do not carry over from k = 1: the result holds on average over the inputs, not for each input, and it bounds the total drift, not the drift of each feature.

2. Alignment

Mean cosine between the two clients' directions for the same feature: 0.13 ± 0.03 (FedAvg) and 0.34 ± 0.04 (FedAvg + MechLeash-FL), mean ± SD, 10/10, p = 0.001.

3. Ablations

AE: our SAE without top-k, i.e., a dense ReLU autoencoder, refitted every round like the SAE. SAE-Frozen: an SAE fitted once after the warm-up round and then kept fixed. Anchor: an L2 penalty that pulls the hidden activations toward those of the model sent by the server at the beginning of the round, divided by the mean squared activation like our leash. Toy MSE and Office-Caltech10 validation MSE (lower is better):

Toy MSEOffice-Caltech10 MSE
FedAvg0.05170.0284
+ MechLeash-FL0.04340.0201
+ AE0.0517 (10/10, 0.001)0.0202 (6/10, 0.46)
+ SAE-Frozen0.0497 (10/10, 0.001)0.0254 (10/10, 0.001)
+ Anchor0.0507 (10/10, 0.001)0.0225 (9/10, 0.003)

Brackets: (seeds in which MechLeash-FL is better than that row, p). The toy column is a rerun, so it differs slightly from Section 7.1.

Why the AE is on par on Office-Caltech10: with only 248 training images, the 256-dimensional hidden layer of the FedAvg model has an effective rank of about 23 dimensions. Its variance is spread as if over 23 equally used directions (effective rank: the exponential of the entropy of the eigenvalue distribution of the activation covariance, which equals d when the variance is spread equally over d directions). A dense autoencoder can then reconstruct the layer as well as a sparse one. Sparsity matters when the model has to store more features than it has dimensions, as in the toy model.

4. Classification

Paper's encoder with a 64-dimensional bottleneck and a linear layer on top. Validation accuracy in %:

FedAvgFedProxMOONDitto (personalized)
lambda_rec0.3110.3
Plain19.018.016.625.4
+ MechLeash-FL20.923.424.725.5
Seeds better, p7/10, 0.01910/10, 0.00110/10, 0.0014/10, 0.46

The gains for FedAvg, FedProx and MOON are significant after Holm correction. In Ditto, the leash acts only on the global model (trained like FedAvg), and the personalized accuracy shows no significant difference. Accuracies are around 20 % because the paper's small encoder is trained from scratch on 65 classes (chance is 1.5 %).

5. More clients

We split each Office-Caltech10 domain into random parts. FedAvg + MechLeash-FL versus FedAvg:

Clients481632
lambda_rec0.30.10.030.1
MSE change-30.4 %-27.6 %-10.5 %-12.5 %
Seeds better, p10/10, 0.00110/10, 0.0019/10, 0.04210/10, 0.001

The 4-client column repeats the paper's setting on new seeds. For 8 to 32 clients, lambda_rec was chosen on three separate seeds; the table shows 10 new seeds. All gains hold after Holm correction. On Office-Home (FedAvg), the accuracy gain is 1.6 percentage points at 8 and 16 clients (10/10 each) and not significant at 24 and 32 clients (0.4 and 0.5 points).

6. Training time

Training time per run on the paper's Office-Caltech10 runs (one A100 GPU): FedAvg 15.1 → 18.6 min (+23 %), FedProx 18.8 → 22.4 (+19 %), MOON 15.9 → 20.7 (+31 %), Ditto 34.0 → 38.3 (+13 %), without → with MechLeash-FL.

7. SAE not fully trained

SAE budget (FedAvg)Upload per round vs. FedAvgValidation MSE reductionSeeds better, p
Paper setting: 20 SAE epochs, refit every round1.55x30.4 %10/10, 0.001
10 / 5 / 2 / 1 SAE epochs1.55x29.8 / 29.4 / 28.2 / 23.0 %10/10, 0.001 each
Refit every 4 / 12 rounds1.14x / 1.05x23.0 / 15.2 %10/10, 0.001 / 7/10, 0.014
Fitted once (SAE-Frozen)about 1x10.4 %7/10, 0.042

Refitting only every r rounds skips the fit in r-1 of r rounds, and clients keep the last averaged SAE in between. The first two rows use new seeds (paper: 29.1 %), the last two use the paper's seeds.

Dear Reviewer,

Thank you again for your review. We uploaded the revised paper on 13 Nov (AoE). It includes the changes described above. If any questions or concerns remain, please let us know and we will gladly address them.

Best regards, The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.