- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
FedTeas uses KD from the round-start global model to guide the SAM perturbation on each client. The subsequent parameter update still uses cross-entropy. The paper also introduces two variants, FedTeasC and FedTeasM, that add control-variate and dual corrections. It studies convergence and generalization, and evaluates the methods on CIFAR-10, CIFAR-100, and Tiny-ImageNet.
Critical strengths
1. The change to SAM is simple and does not require extra communication beyond the corresponding baseline. I also appreciate the comparisons of each variant with the method it builds on, and the inclusion of diagnostics that show both improvements and limitations.
2. The strongest results are for FedTeasM under strong CIFAR-100 heterogeneity. It improves over FedSMOO by 1.77 percentage points at and 1.46 points at .
Critical weaknesses
- Contribution. The contribution appears straightforward and limited: FedTeas modifies FedSAM’s perturbation gradient using distillation, while its variants add existing correction mechanisms. The convergence analysis largely follows FedSAM, and the potentially novel bound lacks a complete derivation.
- Parts of the experimental reporting are unclear. The “Average” column in Tables is not defined. For example in Table 13, if it is the mean over the eight settings , nine of the 15 averages do not match the eight values being averaged. For example, SCAFFOLD should be 62.17, not 61.42, and FedTeasC should be 62.08, not 62.58. Correcting these values reverses their ordering. Tables 15 and 17 also disagree on apparently identical settings: FedTeasM has a minimum loss of 0.686 in one table and 0.501 in the other for CIFAR-10 with . These discrepancies need to be resolved. A clarification would help readers.
The broader claim of consistent improvement also seems too strong. Across the 12 accuracy settings, vanilla FedTeas gains only 0.044 percentage points on average over FedSAM.
- I have concerns about theoretical arguments. In the generalization proof, appears between Eqs. (46) and (47) without a derivation showing how the directional covering ratio enters the PAC-Bayes or union-bound term. Theorem 4 also defines as a volume fraction, whereas Definition 1 and Eq. (41) use an average of covering-number ratios.
Questions for authors
Please also refer to Weakness for the questions. Additionally:
-
Could you clarify how Assumption 3 justifies the stochastic-gradient noise bound used in the last inequality of Lemma B.2 (Eq. 18)? Assumption 3 controls normalized-gradient error, which can be zero even when gradient magnitudes vary and the update variance is positive. Is an additional variance assumption needed, or is there an argument that addresses this distinction?
-
Which experiments use fixed , and which use the dynamic schedule? Please give and the relevant pow/mid values. For FedTeasC, was the server control update scaled by as in Algorithm 3, or as in Eq. (12)?
it would be useful to put KD in the update loss while keeping the teacher and weights the same. This would help distinguish the benefit of guiding the perturbation from that of adding distillation more generally.
2 comments
Q1. Assumption 3 and the update-variance term in Lemma B.2.
Assumption 3 controls differences between normalized stochastic and full gradients, describing the angular error relevant to perturbation generation. Lemma B.2 instead bounds the magnitude of the aggregated stochastic update. The latter extends the analysis from directional perturbation error to update-level second-moment control, the normalized-direction assumption alone is insufficient to bound gradient-magnitude variance. We will make this distinction explicit and state the required stochastic-gradient moment condition for Eq. (18).
Q2. KD schedule, temperature, and FedTeasC normalization.
We use fixed target KD weights and . When dynamic scheduling is enabled, the effective KD strength increases toward this fixed upper level according to
Early training therefore emphasizes local CE-based perturbations, while global-teacher guidance gradually strengthens. We use , , and , the scheduling mechanism is shown in Appendix D.1 and Figure 5(a). We will specify which reported experiments use the fixed weights and which enable the schedule. For FedTeasC, all partial-participation control updates use over the selected clients, as in Eq. (12), the factor in Algorithm 3 is a typographical error and will be corrected.
Q3. Effect of KD placement in perturbation generation and parameter updates.
We thank the reviewer for this suggestion. Appendix F.1 (Table 13) already includes FedCodl, an ERM-based KD baseline using the current global teacher, whereas FedGKD employs an ensemble of historical global teachers.
To further isolate the role of KD placement, we additionally compare inner and outer KD on CIFAR-10 with Dirichlet .:
| Method | Acc. (%) ↑ | Loss ↓ | IQR ↓ |
|---|---|---|---|
| FedTeas (Inner KD) | 79.40 | 0.733 | 0.0236 |
| FedTeas (Outer KD) | 78.59 | 0.731 | 0.0143 |
Inner KD achieves 0.81 percentage points higher accuracy at comparable loss, whereas outer KD exhibits lower loss volatility. This reflects their distinct optimization roles: inner KD guides the -bounded perturbation direction, while outer KD directly regularizes parameter updates.
These results provide additional evidence for the benefit of KD-guided perturbation generation, while illustrating the accuracy–stability trade-off between the two formulations.
All corresponding changes, including the additional experimental comparisons and the theoretical clarifications, will be highlighted in blue in the revised manuscript.
We sincerely thank the reviewer for taking the time to carefully evaluate our manuscript and for providing constructive feedback. We appreciate the detailed comments regarding the identified weaknesses and questions. We address these concerns point by point below and hope that our responses and proposed revisions adequately clarify the key issues raised.
W1. Nature and scope of the contribution.
Our goal is not merely to introduce KD as another correction for client drift. We investigate two related but distinct challenges: heterogeneity-induced client drift and consistency between local sharpness-aware exploration and the global teacher-guided landscape. Section 3, Eqs. (4)–(6), and Remark 1 implement this distinction: the inner perturbation is guided by local CE and current-global-teacher KD, whereas the update at the perturbed point retains the CE objective. Thus, teacher knowledge guides the perturbation geometry rather than simply regularizing parameter updates. FedTeasC/M then examine how this guidance interacts with established variance/dual correction methods for client drift, without claiming those correction mechanisms as new. The loss-landscape comparisons (Figure 3 and Figure 4(c)), Hessian results (Table 4), and directional-alignment analysis (Table 5) provide complementary evidence beyond accuracy.
W2. Numerical inconsistencies in the appendix.
We thank the reviewer for identifying inconsistencies in the appendix tables. We will recheck the original experimental records, recompute the affected averages and loss statistics, and correct the corresponding values in the revised version.
W3. Meaning of CΩ and the generalization analysis.
The original purpose of CΩ is to quantify how KD guidance changes directional covering complexity across clients, not to constrain the number of clients or guarantee a smaller generalization bound. Appendix C.1, Eqs. (39)–(41), characterizes directional alignment and defines Ci as the relative covering-number ratio for client i, with CΩ its client-wise average. Table 5 empirically examines these changes, including the proportion of clients with reduced relative complexity after an additional KD-guided step.
The covering term in the original PAC-Bayesian analysis concerns pre-specified layer-wise spectral-norm configurations, whereas CΩ characterizes data- and teacher-dependent perturbation directions. These are distinct covering objects.
We will therefore remove the direct CΩ factor from Theorem 2 and Appendix C.2. The revised derivation will retain the original layer-norm covering construction and union bound across N clients. Since FedTeas changes the optimization procedure without altering the prescribed hypothesis class, the original PAC-Bayesian framework remains applicable under its prior, spectral-norm, margin, and perturbation-stability conditions.
We will also correct the outdated volume-ratio expression in the restated theorem and retain CΩ as a complementary geometric measure. Table 5 supports the observed directional-covering changes, rather than a direct tightening of the generalization bound.