- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates intermediate-layer supervision in federated learning, analyzes the differences between CE supervision and KD, and proposes a method that adaptively adjusts distillation weights based on training progress, teacher reliability, and prediction discrepancy to improve model performance under heterogeneous data distributions.
Critical strengths
- The paper addresses the problem of intermediate-layer supervision in federated learning, which has certain research significance, and attempts to explain the differences between CE and KD from a gradient analysis perspective.
- The proposed method is relatively simple and can be integrated into different federated learning algorithms without requiring additional communication between clients.
- The experiments cover multiple datasets, federated learning algorithms, and model architectures, and include some ablation studies that provide a certain degree of support for the effectiveness of the method.
Critical weaknesses
- The main contribution of this paper is to use the final classifier to guide intermediate layers through knowledge distillation and adaptively adjust the distillation strength. This addresses a relatively basic setting and problem in FL and the novelty is limited. Moreover, despite devoting substantial space to analyzing the differences between CE and KD, the authors ultimately propose only a heuristic combination of weighting factors, without demonstrating its superiority over simpler properly tuned fixed weights.
- Eq. 6–9 mainly analyze the gradient relationships in the logit space and the finite-sample estimation error of CE, but do not consider the differences in parameter gradients between intermediate branches and the final classifier. They also do not prove that KD has a smaller gradient estimation error than CE, or that gradient alignment can lead to better final performance. Therefore, the theoretical analysis is insufficient to support the core conclusion that KD is more stable and effective than CE.
- The main experiments primarily evaluate the proposed adaptive weighting mechanism by incorporating it into several classical FL methods, without comparing against recent and closely related state-of-the-art approaches. Furthermore, the reported performance improvements are relatively modest, making it difficult to establish the practical significance of the proposed method.
- The paper cites FedLC: Optimizing Federated Learning in Non-IID Data via Label-Wise Clustering (IEEE Access 2023), which primarily addresses Non-IID federated learning through label-wise clustering and client selection. However, the logit-margin mechanism described in the experimental implementation actually corresponds to Federated Learning with Label Distribution Skew via Logits Calibration (ICML 2022), which proposes a different method also named FedLC. This is a serious citation and methodological inconsistency that raises concerns about the correctness and reproducibility of the baseline implementation.
- Lack of experimental validation for the central motivation. The paper repeatedly argues that limited local training data in federated learning exacerbates the shortcomings of intermediate-layer CE supervision, thereby motivating the use of KL-based knowledge distillation. However, the experiments do not vary the number of local samples. Consequently, the experimental results do not directly validate the paper's central motivations.
- Table 4 has already shown that, on CIFAR-100 with , achieves 54.71%, whereas achieves only 52.92%. However, Table 5 still uniformly uses . Without a fixed-weight baseline tuned on a validation set, the advantage of Adaptive may partly come from inappropriate baseline parameters.
Questions for authors
- Eq. 13–16 use the average predicted probabilities, entropy, and JS divergence over the entire local dataset, but the computation frequency is not clearly specified. Are these statistics computed once at the beginning of each local training round, or are they continuously updated at each epoch or mini-batch?
- The authors use the entropy of the average prediction distribution to adjust teacher reliability. However, for Non-IID clients with highly concentrated class distributions, this entropy may be very low even when the teacher's predictions are completely correct, resulting in a significant reduction in the KD weight. How do the authors explain this phenomenon? Does this metric actually confound teacher reliability with the client's class distribution?
- The paper determines the KD weight by multiplying the warm-up factor, teacher reliability, and prediction discrepancy. However, if any of these three factors approaches zero, the KD weight will be significantly suppressed, even when the other factors indicate that distillation may be beneficial. Can the authors explain the necessity of this multiplicative design and its advantages over other simple combination strategies?
- Why do mathematically equivalent settings in Table 4 and Table 5 produce different results? On CIFAR-100 IID, Table 4 reports 69.61% for , whereas Table 5 reports for FedAvg Plain. According to Eq. 10, should reduce to Plain without intermediate supervision. However, the two results differ by 0.69%, while the results for are almost identical in the two tables.
- The paper sets the contrastive learning coefficient of MOON to , making its performance nearly identical to FedAvg and rendering the comparison with MOON of little value. Please explain why such a small coefficient was chosen.