acceptodds
Under review as a conference paper at ICLR 2027

FedTeas: Federated Learning via Teacher Distillation and Sharpness-Aware Minimization

Abstract

Data heterogeneity remains a central challenge in Federated Learning (FL), inducing client drift and often steering optimization toward sharp solutions with poor generalization. While Sharpness-Aware Minimization (SAM) can improve flatness and robustness, locally computed SAM perturbations may become misaligned with the global objective under heterogeneous data. To address this, we introduce FedTeas, a family of three methods: vanilla FedTeas, FedTeasC, and FedTeasM that use global knowledge distillation (KD) as directional guidance for local sharpness-aware optimization. By aligning local perturbations with global predictive knowledge, FedTeas mitigates client drift and yields more consistent, robust global models across communication rounds. Rather than applying KD as a conventional parameter-wise regularizer, FedTeas injects distillation gradients directly into the SAM perturbation step, aligning local perturbations with global predictive knowledge. We prove that vanilla FedTeas retains the convergence rate of FedSAM (seminal approach adapting SAM to FL), while the remaining family members preserve the convergence guarantees of their corresponding optimization frameworks. We establish a -conditioned generalization bound that characterizes the impact of KD-guided perturbation directions on federated generalization. Extensive experiments across heterogeneous federated benchmarks show that FedTeas consistently improves accuracy, robustness, and generalization over state-of-the-art federated optimization and sharpness-aware baselines. Code is available at https://anonymous.4open.science/r/FedTeas-17D1.

Primary area:
optimization
Keywords:
Federated Learning, Sharpness-Aware Minimization, Knowledge Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AKFPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

FedTeas uses KD from the round-start global model to guide the SAM perturbation on each client. The subsequent parameter update still uses cross-entropy. The paper also introduces two variants, FedTeasC and FedTeasM, that add control-variate and dual corrections. It studies convergence and generalization, and evaluates the methods on CIFAR-10, CIFAR-100, and Tiny-ImageNet.

Critical strengths

1. The change to SAM is simple and does not require extra communication beyond the corresponding baseline. I also appreciate the comparisons of each variant with the method it builds on, and the inclusion of diagnostics that show both improvements and limitations.

2. The strongest results are for FedTeasM under strong CIFAR-100 heterogeneity. It improves over FedSMOO by 1.77 percentage points at β=0.1\beta=0.1 and 1.46 points at u=10u=10.

Critical weaknesses
  1. Contribution. The contribution appears straightforward and limited: FedTeas modifies FedSAM’s perturbation gradient using distillation, while its variants add existing correction mechanisms. The convergence analysis largely follows FedSAM, and the potentially novel CΩC_\Omega bound lacks a complete derivation.
  2. Parts of the experimental reporting are unclear. The “Average” column in Tables is not defined. For example in Table 13, if it is the mean over the eight settings , nine of the 15 averages do not match the eight values being averaged. For example, SCAFFOLD should be 62.17, not 61.42, and FedTeasC should be 62.08, not 62.58. Correcting these values reverses their ordering. Tables 15 and 17 also disagree on apparently identical settings: FedTeasM has a minimum loss of 0.686 in one table and 0.501 in the other for CIFAR-10 with u=5u=5. These discrepancies need to be resolved. A clarification would help readers.

The broader claim of consistent improvement also seems too strong. Across the 12 accuracy settings, vanilla FedTeas gains only 0.044 percentage points on average over FedSAM.

  1. I have concerns about theoretical arguments. In the generalization proof, CΩC_\Omega appears between Eqs. (46) and (47) without a derivation showing how the directional covering ratio enters the PAC-Bayes or union-bound term. Theorem 4 also defines CΩC_\Omega as a volume fraction, whereas Definition 1 and Eq. (41) use an average of covering-number ratios.
Questions for authors

Please also refer to Weakness for the questions. Additionally:

  1. Could you clarify how Assumption 3 justifies the stochastic-gradient noise bound L2ρ2σl2L^2\rho^2\sigma_l^2 used in the last inequality of Lemma B.2 (Eq. 18)? Assumption 3 controls normalized-gradient error, which can be zero even when gradient magnitudes vary and the update variance is positive. Is an additional variance assumption needed, or is there an argument that addresses this distinction?

  2. Which experiments use fixed α=(1,0.1)\alpha=(1,0.1), and which use the dynamic schedule? Please give τ\tau and the relevant pow/mid values. For FedTeasC, was the server control update scaled by 1/N1/N as in Algorithm 3, or 1/K1/K as in Eq. (12)?

it would be useful to put KD in the update loss while keeping the teacher and weights the same. This would help distinguish the benefit of guiding the perturbation from that of adding distillation more generally.

2 comments

We sincerely thank the reviewer for taking the time to carefully evaluate our manuscript and for providing constructive feedback. We appreciate the detailed comments regarding the identified weaknesses and questions. We address these concerns point by point below and hope that our responses and proposed revisions adequately clarify the key issues raised.

W1. Nature and scope of the contribution.

Our goal is not merely to introduce KD as another correction for client drift. We investigate two related but distinct challenges: heterogeneity-induced client drift and consistency between local sharpness-aware exploration and the global teacher-guided landscape. Section 3, Eqs. (4)–(6), and Remark 1 implement this distinction: the inner perturbation is guided by local CE and current-global-teacher KD, whereas the update at the perturbed point retains the CE objective. Thus, teacher knowledge guides the perturbation geometry rather than simply regularizing parameter updates. FedTeasC/M then examine how this guidance interacts with established variance/dual correction methods for client drift, without claiming those correction mechanisms as new. The loss-landscape comparisons (Figure 3 and Figure 4(c)), Hessian results (Table 4), and directional-alignment analysis (Table 5) provide complementary evidence beyond accuracy.

W2. Numerical inconsistencies in the appendix.

We thank the reviewer for identifying inconsistencies in the appendix tables. We will recheck the original experimental records, recompute the affected averages and loss statistics, and correct the corresponding values in the revised version.

W3. Meaning of CΩC_{\Omega} and the generalization analysis.

The original purpose of CΩC_{\Omega} is to quantify how KD guidance changes directional covering complexity across clients, not to constrain the number of clients or guarantee a smaller generalization bound. Appendix C.1, Eqs. (39)–(41), characterizes directional alignment and defines CiC_i as the relative covering-number ratio for client ii, with CΩC_{\Omega} its client-wise average. Table 5 empirically examines these changes, including the proportion of clients with reduced relative complexity after an additional KD-guided step.

The covering term in the original PAC-Bayesian analysis concerns pre-specified layer-wise spectral-norm configurations, whereas CΩC_{\Omega} characterizes data- and teacher-dependent perturbation directions. These are distinct covering objects.

We will therefore remove the direct CΩC_{\Omega} factor from Theorem 2 and Appendix C.2. The revised derivation will retain the original layer-norm covering construction and union bound across NN clients. Since FedTeas changes the optimization procedure without altering the prescribed hypothesis class, the original PAC-Bayesian framework remains applicable under its prior, spectral-norm, margin, and perturbation-stability conditions.

We will also correct the outdated volume-ratio expression in the restated theorem and retain CΩC_{\Omega} as a complementary geometric measure. Table 5 supports the observed directional-covering changes, rather than a direct tightening of the generalization bound.

Q1. Assumption 3 and the update-variance term in Lemma B.2.

Assumption 3 controls differences between normalized stochastic and full gradients, describing the angular error relevant to perturbation generation. Lemma B.2 instead bounds the magnitude of the aggregated stochastic update. The latter extends the analysis from directional perturbation error to update-level second-moment control, the normalized-direction assumption alone is insufficient to bound gradient-magnitude variance. We will make this distinction explicit and state the required stochastic-gradient moment condition for Eq. (18).

Q2. KD schedule, temperature, and FedTeasC normalization.

We use fixed target KD weights α1=1\alpha_1=1 and α2=0.1\alpha_2=0.1. When dynamic scheduling is enabled, the effective KD strength increases toward this fixed upper level according to

α2t=α21+exp⁡[−pow(t/T−mid)].\alpha_2^{t}=\frac{\alpha_2}{1+\exp[-\mathrm{pow}(t/T-\mathrm{mid})]}.

Early training therefore emphasizes local CE-based perturbations, while global-teacher guidance gradually strengthens. We use τ=2\tau=2, pow=15\mathrm{pow}=15, and mid=0.25\mathrm{mid}=0.25, the scheduling mechanism is shown in Appendix D.1 and Figure 5(a). We will specify which reported experiments use the fixed weights and which enable the schedule. For FedTeasC, all partial-participation control updates use 1/K1/K over the KK selected clients, as in Eq. (12), the 1/N1/N factor in Algorithm 3 is a typographical error and will be corrected.

Q3. Effect of KD placement in perturbation generation and parameter updates.

We thank the reviewer for this suggestion. Appendix F.1 (Table 13) already includes FedCodl, an ERM-based KD baseline using the current global teacher, whereas FedGKD employs an ensemble of historical global teachers.

To further isolate the role of KD placement, we additionally compare inner and outer KD on CIFAR-10 with Dirichlet β=0.5\beta=0.5.:

MethodAcc. (%) ↑Loss ↓IQR ↓
FedTeas (Inner KD)79.400.7330.0236
FedTeas (Outer KD)78.590.7310.0143

Inner KD achieves 0.81 percentage points higher accuracy at comparable loss, whereas outer KD exhibits lower loss volatility. This reflects their distinct optimization roles: inner KD guides the ρ\rho-bounded perturbation direction, while outer KD directly regularizes parameter updates.

These results provide additional evidence for the benefit of KD-guided perturbation generation, while illustrating the accuracy–stability trade-off between the two formulations.


All corresponding changes, including the additional experimental comparisons and the theoretical clarifications, will be highlighted in blue in the revised manuscript.

Reviewer_bej7OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes FedTeas, a federated learning framework that integrates knowledge distillation (KD) into sharpness-aware minimization (SAM) to mitigate client drift under heterogeneous data. Three variants, FedTeas, FedTeasC, and FedTeasM, are introduced with convergence and generalization analyses. While experiments demonstrate improvements in some settings, concerns remain regarding theoretical correctness and optimization consistency.

Critical strengths
  1. The paper addresses an important problem in federated learning, namely the misalignment between local sharpness-aware optimization and the global objective under non-IID data distributions.
  2. Incorporating knowledge distillation into perturbation generation rather than directly regularizing parameter updates provides an interesting perspective on combining output-space consistency with sharpness-aware optimization.
  3. The proposed framework covers three optimization settings, allowing comparisons with FedSAM, FedGAMMA, and FedSMOO.
  4. The evaluation includes multiple heterogeneous data partitions and additional analyses of gradient alignment, Hessian eigenvalues, and optimization stability.
Critical weaknesses
  1. Insufficient justification for KD-guided perturbations. Eq. (6) constructs perturbations using CE+KD gradients but updates parameters using CE alone. Such perturbations need not maximize the original CE loss and may even oppose its ascent direction. Their claimed sharpness benefits remain theoretically unproven.
  2. Inconsistencies in the convergence analysis. Theorem 1 concerns expected gradient norms at global iterates, whereas Appendix B primarily bounds squared gradient norms at perturbed iterates. These quantities imply different convergence rates, and the transition between them is not justified.
  3. Missing assumptions in the perturbation drift bound. Lemma B.5 relies on smoothness to bound normalized KD-gradient variations. However, L-smoothness cannot control normalized directions near zero gradients without additional assumptions. Eq. (22) also appears inconsistent with the actual CE-based updates.
  4. Inconsistent directional complexity and generalization bound. CΩ is defined using covering-number ratios in the main text but volume ratios in Appendix C.2. Definition 1 also appears incompatible with the CΩ > 1 values in Table 5, raising concerns about the generalization bound.
  5. Inconsistent algorithm definitions and zero-gradient handling. FedTeasC uses 1/N in Algorithm 3 but 1/K in Eq. (12), resulting in different updates under partial participation. Additionally, the KL gradient vanishes when student and teacher initially coincide, potentially causing undefined normalization.
Questions for authors
  1. What exact objective does KD-guided SAM optimize? Under what conditions can its perturbations improve CE sharpness when CE and KD gradients conflict?
  2. How does Theorem 1 follow from the squared-gradient bounds at perturbed iterates, and what additional assumptions justify Lemma B.5?
  3. Which definition of CΩ is intended? How can it explain Table 5's reported values and rigorously support the PAC-Bayesian generalization bound?
  4. Does FedTeasC use 1/N or 1/K for control-variate aggregation? How are zero KL gradients handled during initialization?
2 comments

We sincerely thank the reviewer for taking the time to carefully evaluate our manuscript and for providing constructive feedback. We appreciate the detailed comments regarding the identified weaknesses and questions. We address these concerns point by point below and hope that our responses and proposed revisions adequately clarify the key issues raised.

1. KD-guided perturbations and the optimized objective (W1, Q1).

FedTeas is not designed solely to correct client drift. It additionally investigates how teacher guidance can promote consistency between local sharpness-aware exploration and the global predictive objective. Section 3 (Eqs. (4)–(6), Remark 1) uses the compound CE+KD objective to construct the inner perturbation while retaining CE for the outer update. Consequently, the student explores adverse directions reflecting both local supervision and the cost of deviating from the global teacher, rather than independently maximizing each loss. This is a teacher-guided search direction, not a claim that the perturbation always maximizes CE sharpness. Under opposing CE and KD gradients, CE ascent is not guaranteed, our objective is the alignment of the local sharpness-aware search with teacher-informed global consistency, rather than unconditional improvement of CE sharpness. Figure 4 and Tables 4–5 provide complementary alignment and curvature observations, without establishing such a universal guarantee.

MethodAcc. (%) ↑Loss ↓IQR ↓
FedTeas (Inner KD)79.400.7330.0236
FedTeas (Outer KD)78.590.7310.0143

Effect of KD placement. The additional comparison on CIFAR-10 with Dirichlet β=0.5\beta=0.5 shows that inner KD achieves 0.81 percentage points higher accuracy at comparable loss, while outer KD exhibits lower loss volatility. Inner KD guides the ρ\rho-bounded perturbation search, whereas outer KD directly regularizes parameter updates. These findings demonstrate a placement-dependent accuracy–stability trade-off.

2. From local perturbed updates to global stationarity (W2).

The convergence proof in Appendix B follows the federated SAM descent-analysis framework of Qu et al. (2022) and Fan et al. (2024), adapted to the KD-guided perturbation. Local estimates are first derived for the perturbation-based client updates. Crucially, Lemma B.6 incorporates these local contributions into the inner-product bound involving the global gradient, while Lemma B.7 establishes the descent inequality at the global iterate. Theorem 1 then follows by combining these lemmas and applying the standard summation and step-size arguments, consistent with the cited analyses. Thus, the global stationarity statement is based on the local-to-global descent argument, not on directly identifying local perturbed gradients with gradients at global iterates. We will highlight this existing proof sequence and its references more explicitly in Appendix B.

3. Normalized-gradient drift and the CE update (W3, Q2).

The proof of Lemma B.5 (Appendix B.2, Eqs. (22)–(24)) considers a sufficiently small local learning rate ηl\eta_l to control the one-epoch gradient variation.

Using the Hessian-based first-order expansion and a small angular deviation ϕi,j\phi_{i,j}, the first-order approximation of tan⁡ϕi,j\tan\phi_{i,j} yields the corresponding perturbation-direction drift bound, following the analysis of Qu et al. (2022).

We will explicitly state the required nonzero reference-gradient condition (or an appropriately stabilized normalization) to ensure that the normalized direction is well-defined. The CE-based parameter updates are handled separately through Lemmas B.6 and B.7.

4. Meaning of CΩC_{\Omega} and the generalization analysis (W4, Q3).

We appreciate the reviewer identifying an inconsistency introduced when restating the theorem in Appendix C.2. The intended definition is consistently that of Definition 1 and Eq. (41):

C_{\Omega}=\frac1N\sum_{i=1}^{N}C_i.$$ The purpose of $C_{\Omega}$ is to quantify the relative change in perturbation-direction covering complexity under KD guidance. Table 5 reports Monte Carlo estimates across ten clients. Due to data heterogeneity, the initial ratios are not necessarily below one, an additional KD-guided step increases the observed proportion of clients with $C_i < 1$. We further clarify that the volume-ratio expression in the restated Appendix C.2 theorem is outdated, Definition 1 and Eq. (41) provide the intended definition. The PAC-Bayesian covering term concerns pre-specified layer-wise spectral-norm configurations, while $C_{\Omega}$ describes data-dependent perturbation directions. We will therefore remove the direct $C_{\Omega}$ correction from Theorem 2 and Appendix C.2, retaining the original layer-norm discretization and union bound over $N$ clients under the applicable PAC-Bayesian assumptions. Accordingly, $C_{\Omega}$ remains a complementary geometric measure supported by Table 5, rather than a direct correction to the generalization bound. ### **5. Control-variate scaling and zero-KL initialization (W5, Q4).** Thank you for identifying the transcription error in Algorithm 3: under partial participation, the intended and implemented server control update uses $1/K$, as in Eq. (12), where $K$ is the number of participating clients, *not* $1/N$. We will correct the pseudocode. If the student initially coincides with its teacher and the KL gradient vanishes, the coupled CE+KD perturbation reduces to the CE-only (FedSAM) direction, provided the CE gradient is nonzero. We will explicitly clarify the normalization convention for zero or near-zero gradients in the algorithm. The corresponding zero-gradient handling will be stated consistently with the conditions used in Lemma B.5. In practice, small positive stabilizers are used for gradient normalization, as commonly done in SAM-based implementations (Qu et al., 2022, Sun et al., 2023, Fan et al., 2024). For the decoupled form, a zero KL component must contribute zero rather than be separately normalized. ---- All corresponding changes, including the additional experimental comparisons and the theoretical clarifications, will be highlighted in blue in the revised manuscript.
Reviewer_yWSWOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

FedTeas addresses the mismatch between local sharpness and global flatness under data heterogeneity in federated learning. It uses the global model as the teacher and calculates the local SAM perturbation from a distillation objective, which combines cross-entropy loss and KL divergence. Parameters are then updated via the standard CE loss objective at the perturbed weights. FedTeasC and FedTeasM add variance correction and dual-smoothing techniques, respectively. Finally, this paper presents convergence analysis and a conditional generalization bound, with experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet under Dirichlet and Pathological splits.

Critical strengths
  1. A specific use of global knowledge distillation in perturbation generation. The problem is relevant, where minimizing local sharpness on heterogeneous client data does not necessarily produce global flatness. The distinction in Remark 1 is concrete, which is the distillation objective generates the perturbation, while the standard CE loss objective updates the parameters. This changes how global predictive knowledge enters sharpness-aware optimization. The contribution is an interesting modification of existing federated sharpness-aware methods, although the underlying problem and the correction mechanisms are established.

  2. Experimental evidence under data heterogeneity. Experiments across heterogeneous federated benchmarks show improvements in test accuracy and local-global alignment, with the clearest benefits observed for FedTeasM. These results support the usefulness of KD-guided perturbation when combined with dual correction under strong data heterogeneity.

Critical weaknesses
  1. The experiments do not yet isolate the benefit of the proposed separation between perturbation generation and parameter updates. Section 3 distinguishes FedTeas from KD as a parameter-update regularizer. This distinction is central because combining distillation with sharpness-aware optimization already has studied. For example, SASD in Wu et al. (CVPR 2024, Section 3.2) applies sharpness-aware optimization to the distillation loss. FedTeas differs by using the distillation objective in Eq. (4) for perturbation generation and the CE loss for parameter updates. Its contribution therefore depends on establishing the value of this specific separation. Comparing the objectives would also distinguish the proposed method from earlier sharpness-aware distillation.

  2. The existing FedGKD comparison uses an ensemble of historical global models, whereas FedTeas uses the current global teacher. Figure 4(a) mainly compares FedTeas with decoupled FedTeas. Thus, teacher choice and the role of KD are not fully separated. Therefore, the authors lack of identifying any existing comparison that isolates these effects. If none is available, an additional experiment, if feasible, would compare perturbation generation from Eq. (4) followed by CE parameter updates against perturbation generation from CE followed by parameter updates using Eq. (4), with the same teacher model, local training settings, and correction mechanism.

  3. The corresponding SAM-based baseline without KD is already available. One moderate and one strongly heterogeneous setting would be sufficient. This is a suggestion for testing the central claim, rather than a request to extend the application scope. It would clarify when the proposed separation helps, especially because vanilla FedTeas is slightly below FedSAM on average across the four CIFAR-100 settings, while the larger gains occur for FedTeasM.

  4. The connection between directional covering complexity and the generalization bound needs further justification. The main issue is how the proposed CΩ is considered in the generalization bound. In Appendix C.2, Eq. (46) gives the bound before the federated covering argument. Moreover, lines 1476-1480 then invoke a union bound over clients and a cover of the layer-norm configurations, and then introduce the restriction on perturbation directions by referring to Eq. (41). However, the intermediate bound showing how this directional restriction changes the size of that cover is not provided. In particular, Eq. (41) averages client-level ratios of angular covering numbers. it is not explained why this average supplies the multiplicative factor inside the logarithmic term of Eq. (47). Without this step, it is difficult to assess whether the final bound follows for the quantity measured in Table 5.

  5. There exist some specific inconsistencies in the statement of the quantity used in this paper. For example, Theorem 2 and Appendix C.1, Eqs. (40)-(41), use the average of the relative directional covering complexities. Appendix C.2 restates the theorem but defines CΩ as Vol(Ω(α_1, α_2))/Vol(S^{d-1}) in lines 1424-1426. After Eq. (47), it returns to the client average. This may be an unrevised expression in the restated theorem and does not by itself invalidate the result. Please confirm the definition and correct that expression, or state the assumptions and connecting the two quantities if both are needed.

  6. To support the theoretical claim, please provide the covering-number inequality used to obtain Eq. (47), explain why the client average in Eq. (41) is sufficient for the union bound, and specify how the dependence of the direction sets on the training data is handled. If the available argument only supports the bound before this additional factor, keeping that result and presenting CΩ as an empirical metric would make the scope clear. Table 5 does not contradict a conditional bound, as its reported CΩ estimates exceed one, while Cnext describes a separate update. The interpretation should therefore distinguish the measured improvements in directional alignment from evidence that the CΩ term tightens the generalization bound. This distinction matters because the proposed bound is presented as a theoretical contribution of KD-guided perturbation.

  7. For convergence, Lemma B.5 divides by the initial gradient norm when bounding changes in normalized gradients (pp. 18-19). Please state the conditions that control this denominator and justify their use when perturbation generation and parameter updates have different objectives. The proof of Theorem 1 also chooses a perturbation radius proportional to 1/√T, whereas Table 11 uses fixed radius. Please state the radius requirement explicitly and explain what guarantee, including any residual term, applies to the experimental setting.

Questions for authors
  1. Clarification of the contribution and experimental evidence. Which existing results isolate the effect of using KD for perturbation generation rather than parameter updates, with the same global teacher and correction mechanism? In particular, is such a comparison included in Figure 4(a) or the appendices? This would determine whether the reported gains support the specific separation in Remark 1.

  2. Clarification of the generalization bound. Is the volume-ratio expression in Appendix C.2, lines 1424-1426, an unrevised expression, with Eq. (41) throughout? Using the definition, what covering-number inequality justifies the CΩ factor in Eq. (47), including the use of the client average and the dependence of direction sets on training data? This would clarify whether the proposed bound follows from the preceding argument.

  3. Clarification of the convergence guarantee. What assumptions control the initial gradient norm in the denominator of Lemma B.5, and does the stated convergence rate require a perturbation radius proportional to 1/√T? What guarantee applies to the fixed radii used in Table 11? These conditions determine the scope of the claimed convergence result.

3 comments

We sincerely thank the reviewer for taking the time to carefully evaluate our manuscript and for providing constructive feedback. We appreciate the detailed comments regarding the identified weaknesses and questions. We address these concerns point by point below and hope that our responses and proposed revisions adequately clarify the key issues raised.

1. KD-guided perturbations, teacher choice, and experimental isolation (W1–W3, Q1).

Methodological distinction. FedTeas targets two related but distinct issues in heterogeneous FL: client drift and consistency between local and global sharpness-aware exploration. Section 3, Eqs. (4)–(6), constructs the inner perturbation using the compound objective α1LCE+α2DKL\alpha_1\mathcal L_{\mathrm{CE}}+\alpha_2\mathcal D_{\mathrm{KL}}, while updating model parameters through CE at the resulting perturbed point. The student therefore explores adverse directions jointly informed by local labels and the global teacher, we do not claim that these directions always maximize CE sharpness. This is not simply KD regularization of the outer update.

We appreciate the reference to SASD (Wu et al., CVPR 2024). Its sharpness-aware self-distillation is designed for enhancing a source model using knowledge from a fine-tuned auxiliary model, with sharpness-aware optimization driven by a distillation objective. FedTeas instead addresses federated training from initialization, where supervised CE remains essential and the inner direction is determined by CE+KD. Setting α1=0\alpha_1=0 in our perturbation objective gives a KD-only inner direction, analogous to the distillation-focused perturbation component, but does not make the full training objectives or learning settings identical.

Teacher-controlled comparisons. FedGKD, selected among the stronger KD baselines for the main text, employs an ensemble of historical global teachers. Owing to space constraints, additional distillation baselines are documented in Appendix E (Table 12) and Appendix F.1 (Table 13): in particular, FedCodl provides an ERM-based KD reference using the current global teacher rather than that historical ensemble. It helps distinguish teacher mechanisms, although it is not itself a matched inner/outer SAM objective swap. Figure 4(a) examines KD-weight and decoupling effects, rather than directly serving as that swap.

Direct KD-placement ablation. We therefore additionally compare KD in perturbation generation (Inner) against KD in parameter updates (Outer) on CIFAR-10 with Dirichlet β=0.5\beta=0.5. The observed results are:

KD placementAccuracy (%) ↑Loss ↓IQR ↓
FedTeas (Inner)79.400.7330.0236
FedTeas (Outer)78.590.7310.0143

Inner KD improves accuracy by 0.81 percentage points at comparable loss, whereas outer KD shows lower loss volatility (IQR). This demonstrates a placement-dependent accuracy–stability trade-off in the evaluated setting. Inner KD guides the ρ\rho-bounded, normalized perturbation direction, outer KD regularizes the update objective more directly. A lower IQR supports more stable loss fluctuations, not an unconditional claim of stronger regularization or superior generalization.

Interpreting heterogeneous performance. We agree that vanilla FedTeas does not exceed FedSAM in every CIFAR-100 split. The vanilla formulation isolates teacher-guided perturbation behavior, but, like FedSAM, does not explicitly include a variance- or dual-based client-drift correction. FedTeasC/M combine that direction guidance with existing correction mechanisms, explaining why their benefits can be more apparent under stronger heterogeneity. We assess not only accuracy (Tables 1–2, 13), but also loss and volatility (Table 3, Appendix F.2), loss geometry (Figure 3), Hessian measurements (Table 4), and directional alignment (Table 5).

2. Directional covering complexity and PAC-Bayesian generalization (W4–W6, Q2).

We appreciate the reviewer identifying the gap in the original transition from Eqs. (46) to (47). The intended definition is the client-averaged relative directional covering complexity, as in Definition 1 and Appendix C.1:

\qquad C_{\Omega}=\frac1N\sum_{i=1}^{N}C_i.$$ Its purpose is to quantify how teacher guidance reshapes client-specific perturbation-direction covers, *not* to alter the client count or guarantee that a PAC-Bayesian bound tightens. Appendix C.1 provides the geometric directional analysis, Table 5 further measures this effect through Monte Carlo estimates across ten clients. The initial proportions with $C_i < 1$ are approximately 40–43%, increasing to 66.67–70% after an additional KD-guided step. This does not require every client ratio, or their average, to fall below one, which is consistent with heterogeneous client data. Crucially, the covering term in the original margin-based PAC-Bayesian proof concerns a **pre-specified layer-wise spectral-norm configuration cover**, $C_{\Omega}$ instead describes **data- and teacher-dependent perturbation directions**. These are different covering objects. We will therefore **remove the direct $C_{\Omega}$ factor** from Theorem 2 and Appendix C.2 rather than assume an unsupported mapping between the two. Since FedTeas modifies the optimization procedure without changing the neural architecture or prescribed hypothesis class, the original FedSAM-style PAC-Bayesian framework remains applicable *under its prior, spectral-norm, margin and perturbation-stability conditions*. The step from Eq. (46) to the revised Eq. (47) will retain the original layer-norm discretization and union bound over $N$ clients, without a directional-covering correction. We will also replace the outdated **volume-ratio** expression in the restated appendix theorem with the covering-number definition above and present $C_{\Omega}$ separately as an empirically supported geometric characterization. Table 5 is evidence of **directional-complexity change**, not a direct validation of a tighter PAC-Bayesian bound. This revision also avoids treating the training-data-dependent directional covers as a data-independent hypothesis cover.

3. Convergence conditions, normalized directions, and perturbation radius (W7, Q3).

The purpose of Lemma B.5 is to bound the drift of the KD-guided perturbation direction along CE-based local updates. Appendix B uses a small local step size ηl\eta_l to control the one-epoch gradient variation and a small angular deviation ϕi,j\phi_{i,j}, followed by a first-order expansion of tan⁡ϕi,j\tan\phi_{i,j} (Eqs. (22)–(24)), the approach follows the angular-drift treatment of Qu et al. (2022). We will state the required nonzero reference-gradient (or stabilized-normalization) condition explicitly, since smoothness and a small angle alone do not guarantee a nonvanishing denominator. Lemmas B.6 and B.7 then connect local perturbed update estimates to the global descent inequality, following the analysis framework of Qu et al. (2022) and Fan et al. (2024).

Concerning ρ\rho, the ρ=O(T−1/2)\rho=\mathcal O(T^{-1/2}) choice in the theorem is a horizon-dependent theoretical scaling, where TT is the total communication horizon, it can be selected once before training and need not decrease with the round index tt. Thus using a constant radius within a run is not inconsistent with this construction, as also seen in fixed-radius SAM experiments (Qu et al., 2022, Sun et al., 2023, Fan et al., 2024). In our experiments, ρ\rho remains fixed throughout training. Furthermore, Figure 4(b) evaluates different fixed perturbation radii under the same communication horizon, providing empirical evidence of their influence on model performance. We will clarify this distinction and explicitly retain the ρ\rho-dependent terms in the finite-horizon convergence analysis.


All corresponding changes, including the additional experimental comparisons and the theoretical clarifications, will be highlighted in blue in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.