acceptodds
Withdrawn from ICLR 2027

Rethinking Intermediate Supervision in Federated Learning with Adaptive Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Federated Learning (FL) enables multiple clients to collaboratively train a shared model while keeping their data local. Recent FL methods increasingly introduce supervision at intermediate layers, yet the effects of different forms of supervision in FL remain unclear. We examine Cross-Entropy (CE) and Knowledge Distillation (KD) for intermediate branch supervision in FL. While CE directly uses hard labels, KD provides richer guidance through the final classifier's soft predictions. Our analysis shows that this distinction becomes particularly important under federated local training, where CE is more sensitive to limited local data while KD provides more stable guidance. Motivated by these findings, we adopt KD for intermediate supervision. Its appropriate strength, however, can vary across federated settings and during training, so we propose adaptive KD weighting that adjusts the distillation strength to local training conditions. Experiments across multiple datasets and FL algorithms show that the proposed approach provides consistent performance across diverse federated settings.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Federated Learning, Knowledge Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_raeFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates intermediate-layer supervision in federated learning, analyzes the differences between CE supervision and KD, and proposes a method that adaptively adjusts distillation weights based on training progress, teacher reliability, and prediction discrepancy to improve model performance under heterogeneous data distributions.

Critical strengths
  1. The paper addresses the problem of intermediate-layer supervision in federated learning, which has certain research significance, and attempts to explain the differences between CE and KD from a gradient analysis perspective.
  2. The proposed method is relatively simple and can be integrated into different federated learning algorithms without requiring additional communication between clients.
  3. The experiments cover multiple datasets, federated learning algorithms, and model architectures, and include some ablation studies that provide a certain degree of support for the effectiveness of the method.
Critical weaknesses
  1. The main contribution of this paper is to use the final classifier to guide intermediate layers through knowledge distillation and adaptively adjust the distillation strength. This addresses a relatively basic setting and problem in FL and the novelty is limited. Moreover, despite devoting substantial space to analyzing the differences between CE and KD, the authors ultimately propose only a heuristic combination of weighting factors, without demonstrating its superiority over simpler properly tuned fixed weights.
  2. Eq. 6–9 mainly analyze the gradient relationships in the logit space and the finite-sample estimation error of CE, but do not consider the differences in parameter gradients between intermediate branches and the final classifier. They also do not prove that KD has a smaller gradient estimation error than CE, or that gradient alignment can lead to better final performance. Therefore, the theoretical analysis is insufficient to support the core conclusion that KD is more stable and effective than CE.
  3. The main experiments primarily evaluate the proposed adaptive weighting mechanism by incorporating it into several classical FL methods, without comparing against recent and closely related state-of-the-art approaches. Furthermore, the reported performance improvements are relatively modest, making it difficult to establish the practical significance of the proposed method.
  4. The paper cites FedLC: Optimizing Federated Learning in Non-IID Data via Label-Wise Clustering (IEEE Access 2023), which primarily addresses Non-IID federated learning through label-wise clustering and client selection. However, the logit-margin mechanism described in the experimental implementation actually corresponds to Federated Learning with Label Distribution Skew via Logits Calibration (ICML 2022), which proposes a different method also named FedLC. This is a serious citation and methodological inconsistency that raises concerns about the correctness and reproducibility of the baseline implementation.
  5. Lack of experimental validation for the central motivation. The paper repeatedly argues that limited local training data in federated learning exacerbates the shortcomings of intermediate-layer CE supervision, thereby motivating the use of KL-based knowledge distillation. However, the experiments do not vary the number of local samples. Consequently, the experimental results do not directly validate the paper's central motivations.
  6. Table 4 has already shown that, on CIFAR-100 with β=0.1\beta=0.1, λ=0.1\lambda=0.1 achieves 54.71%, whereas λ=0.3\lambda=0.3 achieves only 52.92%. However, Table 5 still uniformly uses λ=0.3\lambda=0.3. Without a fixed-weight baseline tuned on a validation set, the advantage of Adaptive may partly come from inappropriate baseline parameters.
Questions for authors
  1. Eq. 13–16 use the average predicted probabilities, entropy, and JS divergence over the entire local dataset, but the computation frequency is not clearly specified. Are these statistics computed once at the beginning of each local training round, or are they continuously updated at each epoch or mini-batch?
  2. The authors use the entropy of the average prediction distribution to adjust teacher reliability. However, for Non-IID clients with highly concentrated class distributions, this entropy may be very low even when the teacher's predictions are completely correct, resulting in a significant reduction in the KD weight. How do the authors explain this phenomenon? Does this metric actually confound teacher reliability with the client's class distribution?
  3. The paper determines the KD weight by multiplying the warm-up factor, teacher reliability, and prediction discrepancy. However, if any of these three factors approaches zero, the KD weight will be significantly suppressed, even when the other factors indicate that distillation may be beneficial. Can the authors explain the necessity of this multiplicative design and its advantages over other simple combination strategies?
  4. Why do mathematically equivalent settings in Table 4 and Table 5 produce different results? On CIFAR-100 IID, Table 4 reports 69.61% for λ=0\lambda=0, whereas Table 5 reports 70.3±0.0470.3\pm0.04% for FedAvg Plain. According to Eq. 10, λ=0\lambda=0 should reduce to Plain without intermediate supervision. However, the two results differ by 0.69%, while the results for λ=0.3\lambda=0.3 are almost identical in the two tables.
  5. The paper sets the contrastive learning coefficient of MOON to μ=0.01\mu=0.01, making its performance nearly identical to FedAvg and rendering the comparison with MOON of little value. Please explain why such a small coefficient was chosen.
Reviewer_7HYhOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates cross-entropy and knowledge distillation as intermediate supervision in federated learning. Based on gradient analysis and empirical observations, it proposes an adaptive self-distillation strategy that adjusts the supervision strength according to training progress, teacher reliability, and prediction disagreement between intermediate branches and the final classifier.

Critical strengths

-The paper examines how hard-label supervision and self-distillation affect local training through gradient analysis and complementary empirical diagnostics. The distinction between intermediate-branch accuracy and final-model performance is particularly useful.

-The adaptive weighting strategy uses locally available signals and can be integrated into different FL algorithms. The auxiliary branches are removed after training, leaving the inference architecture unchanged.

Critical weaknesses

The novelty is quite limited. Prior studies [1, 2] have already investigated knowledge distillation and its relationship with cross-entropy supervision in heterogeneous federated learning. Although this paper focuses on intermediate supervision, the conceptual advance over these studies remains unclear. Overall, the work appears to be an incremental extension of existing ideas.

-The additional gains from adaptive weighting over fixed-weight distillation are modest. Moreover, in several non-IID CIFAR-10 and CIFAR-100 settings, the proposed method only matches or slightly underperforms the plain baseline.

-The main experiments are restricted to standard image classification benchmarks, with heterogeneity primarily simulated through label-distribution skew. This evaluation provides limited evidence of effectiveness under realistic domain shifts or more complex learning tasks. I encourage the authors to include more challenging datasets with feature heterogeneity and explore tasks beyond image classification to better demonstrate the method’s generality.

[1] Cd2-pfed: Cyclic distillation-guided channel decoupling for model personalization in federated learning. (CVPR.2022)

[2] Fed-CO2: Cooperation of Online and Offline Models for Severe Data Heterogeneity in Federated Learning. (Neurips.2023)

Questions for authors

What new understanding of CE versus distillation does this work provide, and what makes the proposed approach more than an incremental application of existing distillation principles?

Reviewer_AH1jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper compares CE and KD as supervision objectives for intermediate branches in FL, analyzing their gradient relationships and finite-sample behavior. It proposes KD-only branch supervision with an adaptive weight combining round-wise warm-up, teacher reliability, and prediction discrepancy. Experiments cover four datasets, five FL algorithms, and additional backbones (ResNet50, MobileNetV2).

Critical strengths
  1. The paper analyzes why intermediate supervision should use soft vs. hard targets in FL, rather than just adding auxiliary losses.
  2. Four datasets, five FL algorithms, IID/non-IID settings, three seeds; clear gains on TinyImageNet and ImageNet100-64 with ResNet18.
  3. The weighting uses only local signals, works across FL algorithms, and Appendix F transparently reports overhead; branches are removed at inference.
Critical weaknesses
  1. Section 2 discusses self-distillation, HSD, FedACL, and FedAIMS, but experiments only compare Plain, fixed-weight KD, and the proposed adaptive KD. Without direct comparisons under matched branch architecture and budget, the advance over existing intermediate-supervision designs is unclear.
  2. Eqs. 6-9 show CE has an additional error component aligned with final-CE estimation error, but alignment does not imply larger total error - the covariance between error components matters. Nor does removing one-hot targets eliminate conditional variability, since teacher predictions vary among inputs sharing a representation. A direct CE-vs-KD error comparison with explicit teacher-quality assumptions is needed. The IID finite-sample analysis also lacks a clear connection to the non-IID evaluation setting.
  3. The Fixed baseline uses lambda = 0.3, yet Table 4 shows lambda = 0.1 (54.71%) beats lambda = 0.3 (52.92%) on CIFAR-100 beta = 0.1, and Figure 1 shows adaptation substantially lowers the effective weight under heterogeneity. Comparisons against a tuned fixed weight, warm-up-only, and a shared schedule matched to the adaptive mean weight are needed to separate these effects, along with sensitivity to warm-up duration and concentration parameters.
  4. In Table 5, Adaptive does not beat Plain in several non-IID CIFAR settings (e.g., FedAvg CIFAR-100: 54.8% vs. 55.0%; MOON CIFAR-10: 31.3% vs. 31.7%). In Appendix D, MobileNetV2 Adaptive stays below Plain on ImageNet100-64 under both partitions, and these results use a single seed. Table 10 also reports 1.72x round time over Plain; matched training-time/communication budget evaluations would clarify whether gains justify the overhead.
Questions for authors
  1. What advantage remains over the closest self-distillation and adaptive intermediate-supervision methods under matched architecture and training budget?
  2. Under what assumptions does KD have lower gradient-estimation error than CE? Can this be tested directly at a common checkpoint, varying local sample size and teacher quality?
  3. How much of the gain over Fixed survives with a tuned small KD weight or shared time-dependent schedule? Does client-specific weighting add benefit?
  4. When is intermediate KD worth its overhead, and why is it weaker on MobileNetV2 and non-IID CIFAR settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.