acceptodds
Under review as a conference paper at ICLR 2027

Layer-Selective Sharpness-Aware Minimization for Efficient and Accurate Federated Learning

Abstract

Federated learning (FL) is a distributed learning paradigm that trains a model on distributed private data while preserving data privacy. To improve the global model's generalization, recent studies have increasingly applied Sharpness-Aware Minimization (SAM), known to guide the model toward a flat region in the parameter space, to local training in FL. However, naive application of SAM to local training doubles the computational cost without noticeably improving the validation accuracy of the global model. Previous works have focused on the discrepancy between global and local loss landscapes, however, our study finds that layer-wise sharpness plays a more important role in determining the impact of local SAM on the global model's generalization. In this paper, we propose a general FL framework, FedLSAM that perturbs only a subset of layers with a significant impact on reducing global sharpness, rather than perturbing the entire model. Experimental results show that the proposed method improves generalization performance compared with perturbing all layers, while having a negligible additional computational cost.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Federated Learning, Sharpness-Aware Minimization, Efficient Perturbation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WU5dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FedLSAM, a method that aims to reduce the computational cost of Sharpness-Aware Minimization (SAM) while improving the global model’s generalization performance in non-IID federated learning. At each communication round, the server selects the top-k layers with the largest global update norms, computed from the difference between consecutive global models. Clients perturb only the selected layers using local gradients, then update all model parameters before server aggregation. The paper analyzes the convergence properties of FedLSAM and evaluates it on four datasets. The results show that FedLSAM achieves comparable or superior performance to existing baselines in multiple settings.

Critical strengths
  1. The paper addresses a practically relevant problem. Existing federated SAM methods primarily focus on estimating perturbation directions, while this work investigates which layers should be perturbed.
  2. The algorithm is simple and can be readily integrated into existing federated training workflows.
Critical weaknesses

1、The additional contribution over existing Layer-wise SAM methods is unclear. Prior work already selects the top-k layers for perturbation based on layer-wise gradient norms. This paper retains that basic structure, primarily replacing the selection procedure with server-side selection of a common layer set based on global updates. 2、The mechanism underlying the improvement in global generalization is insufficiently analyzed. The paper does not adequately explain how layer selection based on global updates mitigates the discrepancy between local and global flatness. The experimental results alone may be insufficient to substantiate this mechanism. 3、The experimental evaluation does not sufficiently support the claimed computational efficiency. The paper argues that FedLSAM reduces computational cost because the selected layers lie mostly on the output side, allowing the first backward pass to terminate early. However, it does not report the layers actually selected during training, the extent of the backward computation, or the per-round training time of different methods.

Questions for authors

1、 Please further explain how perturbing only a subset of layers in FedLSAM improves the global model’s generalization performance. 2、 Since layers differ in their number of parameters, directly comparing the L2L_2 norms of their global updates may favor layers with more parameters. Analyze how layer parameter counts affect layer selection. 3、 Reporting which layers are selected, how far the ascent-step backpropagation proceeds, and the per-round computation time relative to baselines could further strengthen the computational efficiency claim.

1 comment

We thank the reviewer for the comments. We would like to address the reviewer’s concerns as follows.

Q1. Please further explain how perturbing only a subset of layers in FedLSAM improves the global model’s generalization performance.

FedLSAM selects layers with large aggregated global updates, so all participating clients perturb the same layers associated with substantial changes to the global model. This may concentrate SAM’s regularization on useful directions while avoiding perturbations in layers that add client-specific noise. Table 5 supports the selection rule: top-k outperforms bottom-k and random selection, while Table 4 shows that perturbing more layers does not always improve accuracy. We will clarify that this is an empirically supported explanation, not a guarantee that update norm directly measures global sharpness.

Q2. Since layers differ in their number of parameters, directly comparing the norms of their global updates may favor layers with more parameters. Analyze how layer parameter counts affect layer selection.

We agree that parameter count can affect update norms, but it does not determine layer selection. In the CIFAR-100/ResNet-18 experiment, some later layers have more parameters yet are not selected in every round. We will add layer parameter counts to the selection-frequency analysis to make this relationship clearer.

Q3. Reporting which layers are selected, how far the ascent-step backpropagation proceeds, and the per-round computation time relative to baselines could further strengthen the computational efficiency claim.

We agree that these measurements would strengthen the efficiency claim. In the CIFAR-100/ResNet-18 experiment, later layers are selected more frequently, allowing the ascent-step backpropagation to stop at the earliest selected layer. We will report the layer-selection frequencies, the extent of this backward pass, and measured per-round times relative to the baselines.

The following shows how many times each layer was selected in the CIFAR-100/ResNet-18 experiment.

Layer Index | Number of Selections

 1      |          5
 2      |          1
 3      |          1
 4      |          1
 5      |          0
 6      |          4
 7      |          2
 8      |          1
 9      |          0
10      |          0
11      |          9
12      |         101
13      |          0
14      |         107
15      |          61
16      |         468
17      |         978
18      |          22
19      |         423
20      |         821
21      |         991
Reviewer_58FUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how to reduce the computational overhead of Sharpness-Aware Minimization (SAM) in Federated Learning (FL) while improving the generalization performance of the global model. The authors propose FedLSAM, whose core idea is: instead of applying SAM perturbation to the entire model, it selects a small number of key layers to perturb based on the magnitude of global model updates across adjacent communication rounds at the server. Experimental results show that FedLSAM improves generalization performance compared with perturbing all layers, while having a negligible additional computational cost.

Critical strengths

The paper investigates the computational overhead introduced by applying SAM to federated local training. This is an important problem because federated clients may have limited computational resources.

The paper achieves promising results across multiple federated benchmarks, includes convergence analysis, and considers systems-level efficiency.

Critical weaknesses

Existing studies have focused on local parameter perturbation and layer-wise SAM. FedLSAM mainly combines these ideas with global update information in federated learning. This design has engineering and application value, but the degree of technical innovation needs to be improved.

The theoretical derivation in the paper is based on global gradient norms, but the actual algorithm uses global parameter update norms to select the Top-k layers. In Non-IID federated learning environments, the two do not necessarily have a stable correlation. The paper lacks a relevant description proving that update norms can accurately reflect the contribution of each layer to global sharpness.

Questions for authors

Q1:Beyond applying existing layer-wise or selective SAM techniques to federated learning, what fundamental methodological contributions does FedLSAM provide? Did the authors compare FedLSAM with direct federated adaptations of ESAM, SAM-ON, or layer-wise SAM under matched training settings?

Q2:The theoretical derivation motivates selecting layers with larger global gradient norms, whereas FedLSAM uses global parameter update norms. Please provide sufficient theoretical or experimental evidence.

1 comment

We thank the reviewer for the comments. We would like to address the reviewer’s concerns as follows.

Q1:Beyond applying existing layer-wise or selective SAM techniques to federated learning, what fundamental methodological contributions does FedLSAM provide? Did the authors compare FedLSAM with direct federated adaptations of ESAM, SAM-ON, or layer-wise SAM under matched training settings?

Prior selective SAM methods were developed primarily in centralized settings to reduce computational cost, whereas FedLSAM is designed for federated learning. FedLSAM selects layers using the previous round’s aggregated global update norms and perturbs those layers during local training. This selection rule is designed to account for heterogeneous client data, and our experiments show improved generalization performance. Among the federated SAM methods considered in this paper, FedLSAM is distinct in combining layer-wise perturbation with no additional model-sized communication or persistent storage overhead.

We have not directly compared FedLSAM with federated adaptations of the prior selective SAM methods. Because those methods were not designed around federated data heterogeneity, replacing local SAM with them may reduce computation without necessarily improving the global model’s generalization. We will add direct comparisons in the revision to evaluate this possibility.

Q2:The theoretical derivation motivates selecting layers with larger global gradient norms, whereas FedLSAM uses global parameter update norms. Please provide sufficient theoretical or experimental evidence.

We agree that the current analysis does not establish that global update norms always rank layers in the same order as global gradient norms. The gradient norm in §3.2 motivates which layers may matter for the first-order increase in global loss. Because the server does not directly compute the global gradient during federated training, FedLSAM instead uses the aggregated model update available from the preceding round. With multiple local steps, SAM perturbations, and heterogeneous client data, this update is only a proxy, not an exact gradient.

Table 5 provides empirical support for the selection rule: choosing layers with large update norms outperforms bottom-kk and random layer selection. It does not directly verify the relationship between update norms, global gradient norms, and layer-wise global sharpness. We will state this limitation explicitly and add diagnostic measurements of whether layers selected by update norm produce larger increases in aggregated global loss under layer-restricted perturbations. These measurements are for evaluating the selection rule; computing global gradients will not become a requirement of FedLSAM training.

Reviewer_2fLKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a general FL framework, FedLSAM that perturbs only a subset of layers with a significant impact on reducing global sharpness, rather than perturbing the entire model. Experimental results show that the proposed method improves generalization performance.

Critical strengths
  • The paper addresses a well motivated problem.

  • The convergence analysis is provided in the paper.

Critical weaknesses
  • The method feels incremental, because the Layer-selective SAM has already been in other literature, such as Lee 2024 KDD, ESAM 2021, and SAM-ON 2023.

  • Table 2 and 3 show the experimental results compared to baseline methods. FedLSAM does not show as the strongest method, as FedWMSAM's performance is almost the same as FedLSAM's. FedWMSAM's performance on FEMNIST is even higher.

  • The impact of the hyperparameter k needs to be further discussed.

Questions for authors

See comments.

1 comment

We thank the reviewer for the comments. We would like to address the reviewer’s concerns as follows.

Q1. The method feels incremental, because the Layer-selective SAM has already been in other literature, such as Lee 2024 KDD, ESAM 2021, and SAM-ON 2023.

A. We agree that perturbing only a subset of parameters or layers was introduced in prior work, including Lee (2024), ESAM, and SAM-ON. Our contribution is the layer-selection rule for federated learning: the server selects a shared set of layers using the previous round’s aggregated global update norms, and all participating clients perturb that set during local training. This uses an aggregate signal in the presence of heterogeneous client data while requiring only the transmission of a few layer indices. Compared with the federated SAM baselines considered in our paper, FedLSAM applies layer-selective perturbation without an additional model-sized communication payload or persistent model-sized state. We will clarify this distinction and avoid presenting selective perturbation itself as novel.

Q2. Table 2 and 3 show the experimental results compared to baseline methods. FedLSAM does not show as the strongest method, as FedWMSAM's performance is almost the same as FedLSAM's. FedWMSAM's performance on FEMNIST is even higher.

A. We agree that the two methods achieve similar accuracy in several settings and that FedWMSAM performs better on FEMNIST. FedLSAM nevertheless obtains numerically higher accuracy in every other setting in Tables 2 and 3. The largest difference is on CIFAR-100, where FedLSAM exceeds FedWMSAM by 3.07%. The methods also differ in communication cost: Table 1 estimates 2M per round for FedLSAM and 3M for FedWMSAM, which transmits an additional momentum vector. We therefore view FedLSAM as a competitive alternative that achieves strong accuracy without that additional model-sized transfer, rather than claiming a substantial accuracy advantage in every setting.

Q3. The impact of the hyperparameter k needs to be further discussed.

A. Table 4 shows that accuracy initially improves as the number of perturbed layers increases, but declines beyond a certain point. The best value differs by setting: k for CIFAR-10/VGG-11 and 8 for CIFAR-100/ResNet-18 among the values evaluated. In the current experiments, we select k for each setting and keep it fixed throughout training. We will discuss this sensitivity more explicitly. The number of layers needed for effective gradient-norm penalization may also change across communication rounds; as future work, we plan to incorporate layer-wise gradient variance into the selection rule and adapt k during training.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.