- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes FedLSAM, a method that aims to reduce the computational cost of Sharpness-Aware Minimization (SAM) while improving the global model’s generalization performance in non-IID federated learning. At each communication round, the server selects the top-k layers with the largest global update norms, computed from the difference between consecutive global models. Clients perturb only the selected layers using local gradients, then update all model parameters before server aggregation. The paper analyzes the convergence properties of FedLSAM and evaluates it on four datasets. The results show that FedLSAM achieves comparable or superior performance to existing baselines in multiple settings.
Critical strengths
- The paper addresses a practically relevant problem. Existing federated SAM methods primarily focus on estimating perturbation directions, while this work investigates which layers should be perturbed.
- The algorithm is simple and can be readily integrated into existing federated training workflows.
Critical weaknesses
1、The additional contribution over existing Layer-wise SAM methods is unclear. Prior work already selects the top-k layers for perturbation based on layer-wise gradient norms. This paper retains that basic structure, primarily replacing the selection procedure with server-side selection of a common layer set based on global updates. 2、The mechanism underlying the improvement in global generalization is insufficiently analyzed. The paper does not adequately explain how layer selection based on global updates mitigates the discrepancy between local and global flatness. The experimental results alone may be insufficient to substantiate this mechanism. 3、The experimental evaluation does not sufficiently support the claimed computational efficiency. The paper argues that FedLSAM reduces computational cost because the selected layers lie mostly on the output side, allowing the first backward pass to terminate early. However, it does not report the layers actually selected during training, the extent of the backward computation, or the per-round training time of different methods.
Questions for authors
1、 Please further explain how perturbing only a subset of layers in FedLSAM improves the global model’s generalization performance. 2、 Since layers differ in their number of parameters, directly comparing the norms of their global updates may favor layers with more parameters. Analyze how layer parameter counts affect layer selection. 3、 Reporting which layers are selected, how far the ascent-step backpropagation proceeds, and the per-round computation time relative to baselines could further strengthen the computational efficiency claim.
We thank the reviewer for the comments. We would like to address the reviewer’s concerns as follows.
Q1. Please further explain how perturbing only a subset of layers in FedLSAM improves the global model’s generalization performance.
FedLSAM selects layers with large aggregated global updates, so all participating clients perturb the same layers associated with substantial changes to the global model. This may concentrate SAM’s regularization on useful directions while avoiding perturbations in layers that add client-specific noise. Table 5 supports the selection rule: top-k outperforms bottom-k and random selection, while Table 4 shows that perturbing more layers does not always improve accuracy. We will clarify that this is an empirically supported explanation, not a guarantee that update norm directly measures global sharpness.
Q2. Since layers differ in their number of parameters, directly comparing the norms of their global updates may favor layers with more parameters. Analyze how layer parameter counts affect layer selection.
We agree that parameter count can affect update norms, but it does not determine layer selection. In the CIFAR-100/ResNet-18 experiment, some later layers have more parameters yet are not selected in every round. We will add layer parameter counts to the selection-frequency analysis to make this relationship clearer.
Q3. Reporting which layers are selected, how far the ascent-step backpropagation proceeds, and the per-round computation time relative to baselines could further strengthen the computational efficiency claim.
We agree that these measurements would strengthen the efficiency claim. In the CIFAR-100/ResNet-18 experiment, later layers are selected more frequently, allowing the ascent-step backpropagation to stop at the earliest selected layer. We will report the layer-selection frequencies, the extent of this backward pass, and measured per-round times relative to the baselines.
The following shows how many times each layer was selected in the CIFAR-100/ResNet-18 experiment.
Layer Index | Number of Selections