acceptodds
Under review as a conference paper at ICLR 2027

Improved Convergence and Generalization Analysis for Online Knowledge Distillation

Abstract

Online knowledge distillation enables distributed models to learn collaboratively without requiring a pretrained teacher. However, the evolving and mutually dependent distillation targets complicate both convergence and generalization analysis. We study online distillation over decentralized networks with heterogeneous model architectures and personalized local distributions. We establish a finite-round convergence bound by controlling objective drift through the neighbors' local gradients and stochastic noise. This analysis does not rely on uniform bounds on gradients or objective drift used in existing analyses and yields tighter guarantees in wide-range parameter regimes. By combining our convergence and generalization analyses, we bound the personalized test risks of the local models produced after a finite number of training rounds. The resulting bound explicitly characterizes the roles of distribution discrepancies, dataset sizes, the trade-off between original supervised and distillation losses, and ensemble weights to construct teachers, including how these effects propagate through the network. This explicit parameter dependence enables tunable-parameter optimization based on a surrogate of the generalization bound. Using ensemble weigh optimization as example, our experiments demonstrate that this approach can substantially improve average personalized test accuracy across a range of heterogeneous settings.

Primary area:
optimization
Keywords:
Online knowledge distillation, convergence bound, generalization bound, ensemble weights

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_g62iOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies online knowledge distillation over decentralized networks of heterogeneous models and personalized data distributions. Unlike offline distillation with a fixed pretrained teacher, each node’s teacher here is a weighted average of its neighbors’ predicted probabilities, using fixed weights and an optional self-weight, evaluated at the round-start parameters and refreshed every communication round. The teachers therefore evolve together with the students, coupling the per-node distillation objectives and making them time-varying. The authors claim three main contributions. First, they propose a finite-round convergence bound (Theorem 1) that controls objective drift via neighbors’ local gradients and stochastic noise, avoiding the uniform bounds on expected stochastic-gradient norms used by Tan et al. (2022) and on round-to-round objective drift used by Abourayya et al. (2025a). They provide a counterexample (Appendix C.5) intended to show that their assumptions do not imply the uniform gradient bound, and necessary/sufficient conditions (Corollary 1 and Theorem 4) under which their bound is tighter than existing bounds. Second, they combine this convergence analysis with a generalization analysis to bound the expected personalized test risk of the finite-round output models (Theorem 5), with explicit dependence on distribution discrepancies, dataset sizes, the trade-off factor, and the ensemble weight matrix, including how these effects propagate through the network. Third, they propose GBOW, an ensemble-weight optimization method derived from a surrogate of this bound. The paper reports experiments on MNIST, FashionMNIST, and CIFAR-10 across five heterogeneity levels (20 nodes; complete graph by default), where GBOW attains the highest best-over-ρ accuracy in 14 of 15 settings against seven weight-design baselines, along with additional experiments on heterogeneous architectures and sparse communication graphs. Proofs are provided in Appendix C, including an approximation and estimation error analysis for the label-marginal proxies used by GBOW (Appendix C.9); experimental details are in Appendix D.

Critical strengths

1.The paper presents the first end-to-end convergence-and-generalization analysis whose bound applies to the actual finite-round training output. Unlike the closest prior analyses---which bound idealized ERM ensembles (Cho et al., 2023), teacher-dependent quantities with no explicit parameter dependence (Shao et al., 2024), or convex KRR estimators (Park et al.) Theorem 5 bounds the expected personalized test risk of the models actually produced by the coupled, time-varying online-distillation procedure, with explicit dependence on the ensemble weights and the trade-off factor. This is the strongest consideration supporting acceptance: it is the paper's genuinely novel contribution and the foundation for the bound-guided weight design (GBOW) underlying the reported empirical gains.

2.The improvement claims are rigorously qualified rather than merely asserted. The paper shows, by a constructive counterexample, that its assumptions do not imply the uniform gradient bound of Tan et al. (2022) (Appendix C.5), and characterizes the claim of a tighter bound through an exact necessary-and-sufficient condition (Lemma 8), a necessary condition (Corollary 1), a sufficient condition (Theorem 4), and a monotonicity analysis (Appendix C.4). For an improvement-oriented paper, the credibility of the improvement claim is central.

Critical weaknesses

1.As presented, the key improvement claim is established only under conditions and is not yet quantified. The conditions in Lemma 8, Corollary 1, Theorem 4, and Appendices C.4/C.6 involve constants that are not directly observable in practice, and the paper does not include an instance showing that the conditions can be met or how large the improvement is. Consequently, it is difficult for a reader to assess whether the promise of "tighter guarantees in wide-range parameter regimes" is non-vacuous in any concrete setting—the consideration that most limited my rating, though a single worked example would likely address it.

Questions for authors

Can you provide one concrete instance---e.g., a simple model where LL, LdL_d, LhL_h, CsgdC_{\mathrm{sgd}}, and UU can all be instantiated, demonstrating that ΦK(Q)<ΦK(Q~)\Phi_K(Q) < \Phi_K(\widetilde{Q}), including the size of the gap as a function of the problem parameters?

2 comments

W1 and Q1: Example with quantified constants and bounds. We thank the reviewer for their excellent suggestion. In the revised manuscript, we have added a fully specified two-node example in Appendix D.8 to quantify the theoretical constants and the corresponding performance improvement.

In particular, we show that this example gives $ L=1,\qquad L_h=\frac{\sqrt2}{2},\qquad L_d=\sqrt2, \qquad \sigma_{\mathrm{sgd}}=C_{\mathrm{sgd}}=0, \qquad U=\rho, \text{ and } \widetilde U=8(1-\rho)\rho/27. $ We then derive closed-form expressions for all three convergence bounds and exact conditions for their comparison as functions of ρ\rho and the number of rounds TT. We also numerically evaluate the actual stationarity quantity S0S_0 and report it alongside these bounds for T∈{10,100,1000,10000}T\in\{10,100,1000,10000\}. For example, at ρ=0.9\rho=0.9 and T=1000T=1000, we have $ S_0\approx0.000781,\qquad Q\approx0.04246,\qquad \widetilde Q\approx0.18125,\text{ and } Q_3\approx0.05458. $ Thus, our bound in Theorem 1 is 76.57%76.57\% smaller than the bound in Theorem 2 and 22.21%22.21\% smaller than that in Theorem 3. Moreover, the derived conditions give Q<Q~Q<\widetilde Q for every integer T≥2T\ge2 and Q<Q3Q<Q_3 for every integer T≥71T\ge71.

Thank you for adding the worked example in Appendix D.8. Taking the reported derivations at face value, the example does what I asked for. It instantiates the relevant constants and shows the resulting gap so my W1/Q1 is addressed and I have no further questions on it.

The example resolves the specific concern I raised but it does not alter my overall assessment of the paper's contributions. I keep my rating at 3, weak acceptance.

Reviewer_1UmjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the convergence and generalization of online knowledge distillation in decentralized networks with heterogeneous models and personalized data distributions. First, it derives an improved finite-round convergence bound by controlling objective drift through neighboring models' gradients and stochastic noise, avoiding restrictive uniform boundedness assumptions. Second, it establishes a network-dependent generalization bound for the trained models, explicitly characterizing the effects of data heterogeneity, dataset sizes, distillation trade-off factors, and ensemble weights. Based on that, the authors propose a generalization-bound-guided ensemble weighting strategy (GBOW). Experiments on MNIST, FashionMNIST, and CIFAR-10 demonstrate its effectiveness across various heterogeneous settings.

Critical strengths

(1) The paper presents a meaningful theoretical contribution by controlling the drift of time-varying distillation objectives through neighboring models' optimization dynamics rather than relying on uniform bounds on stochastic-gradient norms or objective drift.

(2) The generalization bound explicitly characterizes the effects of ensemble weights, including both direct teacher–student interactions and indirect information propagation through the network, providing an interpretable theoretical perspective on teacher selection and motivates a practical weight optimization strategy.

Critical weaknesses

(1) The generalization analysis considers an output iterate selected by minimizing the expected average squared gradient norm across training rounds. However, Algorithm 1 returns the models after the final round, while the experiments report accuracy averaged over the final 1,000 rounds. These output-selection mechanisms are not equivalent. Consequently, the finite-round personalized generalization guarantee does not directly characterize the models evaluated in the experiments.

(2) The proposed GBOW method approximates input-distribution discrepancies using empirical label-marginal discrepancies. Although Appendix C.9 analyzes the approximation error, the quality of this proxy depends on the similarity of class-conditional input distributions, which may not hold under more general domain shifts. The experiments primarily simulate heterogeneity through different class proportions, a setting particularly favorable to label-marginal statistics. It remains unclear how robust the proposed weighting strategy is when nodes have similar label proportions but substantially different input distributions.

Questions for authors

Please address the above problems.

3 comments

We thank the reviewer for their detailed comments. The following are our response to the questions raised.

W1: Output-selection mechanisms. First, we emphasize that our analysis is not tied to any particular output-selection rule. Theorem 1 bounds average stationarity along the training trajectory, independently of how the output iterate is subsequently selected. Equation (63) provides a generalization bound for an arbitrary selected iterate in terms of its corresponding optimization residual. The minimum-expected-gradient selector is used in Eq. (64) only to obtain a particular explicit bound on this residual. This selector is not required for the preceding convergence or residual-dependent generalization bounds. Other output rules can also be accommodated through their corresponding residual bound.

In particular, a commonly used output-selection mechanism in non-convex stochastic optimization is randomized iterate selection [1]-[3]. Define τ∼Unif({T−m+1,…,T})\tau\sim\text{Unif}(\{T-m+1,\ldots,T\}), where 1≤m≤T1 \leq m \leq T. The output of the random iterate selection rule is θ^:=θτ\hat{\boldsymbol\theta}:=\boldsymbol\theta^\tau. As shown below, it renders the output rule, its guarantee, and its empirical evaluation consistent.

Let A(t)A(t) denote the average personalized empirical test accuracy at θt\boldsymbol\theta^t, as defined in (102) of the revised manuscript (originally (98)). Conditional on the training trajectory F\mathcal{F}, we have Eτ[A(τ)∣F]=1m∑t=T−m+1TA(t)\mathbb E_\tau[A(\tau)\mid\mathcal{F}] = \frac1m\sum_{t=T-m+1}^{T}A(t). \textbf{Thus, each run’s window average exactly computes the conditional expected empirical accuracy over the output-selection randomness.} Averaging these scores across independent runs further estimates the expectation over the run randomness. In particular, the existing final-1000-round scores evaluate this randomized output with m=1000m=1000.

Next, we show the generalization guarantee for the randomized output. Define R:=(EF,τ[Ri(θiτ)])i=1N\boldsymbol{R} := ( \mathbb E_{\mathcal{F},\tau} \left[R_i(\theta_i^\tau)\right])_{i=1}^{N}. We have with probability at least 1−δ1-\delta over the training and reference datasets, \begin{align*} \boldsymbol{R} \preceq{} (I-qW)^{-1} (\boldsymbol\epsilon+2\rho\boldsymbol{\Re}) + (I-q^2W^2)^{-1}\boldsymbol{B} + \frac{q+r}{\rho}\mathbf1 + \frac{N(T+1)\Phi_K(Q)}{2m} \boldsymbol{\lambda}. \end{align*} We have added Appendix C.9 to show the details. Moreover, we additionally report results for uniform selection from the final half of the training rounds in Appendix D.6 of the revised manuscript.

References

[1]\text{[1]} Ghadimi, Saeed, and Guanghui Lan. "Accelerated gradient methods for nonconvex nonlinear and stochastic programming." Mathematical Programming (2016): 59-99.

[2]\text{[2]} Zhang, Junyu, and Lin Xiao. "A stochastic composite gradient method with incremental variance reduction." Advances in Neural Information Processing Systems (2019).

[3]\text{[3]} Ghadimi, Saeed, and Guanghui Lan. "Stochastic first-and zeroth-order methods for nonconvex stochastic programming." SIAM journal on optimization (2013): 2341-2368.

W2: Experiments beyond empirical label-marginal discrepancies.: We thank the reviewer for this suggestion. In the revised manuscript, we have added a five-class FashionMNIST experiment by merging consecutive pairs of the original ten classes. The 20 nodes form two groups with identical, uniform label proportions but different within-class input mixtures, controlled by shift strengths 00, 0.30.3, and 0.80.8. The reference set is also label-balanced. Thus, all empirical label-marginal discrepancies supplied to GBOW are exactly zero, even when the conditional input distributions differ substantially. Appendix D.6 details the construction and evaluation. GBOW achieves the highest reported mean for most tested values of ρ\rho at shift strengths 00 and 0.30.3. At the strongest shift, COMET achieves the highest mean, while GBOW ranks second at most tested values of ρ\rho. Comparing their best observed means over ρ\rho, GBOW trails COMET by 0.210.21 percentage points for the final-1,000-round average. These results support GBOW’s competitiveness under substantial conditional input shift.

Regarding the theoretical analysis, the input-discrepancy-based bound (5) does not require the conditional input distributions PX∣Y(i)P_{X\mid Y}^{(i)} to be identical or close across nodes. In the original manuscript, Appendix C.9 (C.10 in the revised version) provides an analysis on the effect of the approximation and finite-sample estimation error terms in the proxy-based bound (96). Under substantial conditional input shift, the approximation error can be strong, making the proxy-based bound (96) less informative.

Thanks for the authors' replies. The rebuttal addresses the concerns regarding theoretical clarifications and additional experiments. The concerns regarding the generality and practical applicability of the proposed surrogate remain unsolved. Under more general distribution shifts, the effectiveness of the proposed label-marginal-based surrogate remains insufficiently established. In particular, under strong conditional input shift, GBOW is outperformed by COMET, and the authors acknowledge that the proxy-based generalization bound may become less informative. Therefore, it is unclear whether the proposed surrogate can reliably guide ensemble-weight optimization beyond label-distribution heterogeneity.

Thank you for the follow-up.

At first, we would like to clarify the empirical evidence for GBOW’s usefulness beyond label-proportion heterogeneity.

The original d=10d=10 experiments have provided evidence of strong superiority of GBOW when label proportions are identical across nodes (Tables 1, 3-6, and 8-11), where all label-discrepancy estimates supplied to GBOW are exactly zero, while the within-class input mixtures can still differ.

In the new experiment, for the final-1,000-round metric, GBOW has the highest reported mean at 11 of the 12 tested values of ρ\rho under shift strength 0.30.3. Under shift strength 0.80.8, COMET leads, but GBOW still ranks second at 10 of 12 values. The respective best-over-ρ\rho means are 83.5383.53% for GBOW and 83.7483.74% for COMET, a gap of 0.210.21 percentage points. Thus, the strongest tested shift reduces GBOW’s relative advantage, but the results still provide evidence of its useful weight choices. There is also a difference in the implementation requirements. The COMET implementation requires a central server to collect and cluster predictions consistently during the whole training process while GBOW’s weight matrix can be computed before the training and without a central server.

GBOW remains effective in these identical label-proportion cases since the considered surrogate includes sample-size-dependent sampling penalties and their propagation through the network, in addition to distribution-discrepancy terms. When the label-discrepancy estimates vanish, these remaining terms still depend on the ensemble weights and provide a basis for their optimization. So the other information retained in the objective can still support useful weight selection.

Secondly, our analysis and broader weight-optimization framework do not depend on the use of label-marginal proxies. We propose this label-marginal-proportion-based method to avoid exchanging raw private inputs for input-discrepancy estimation for practical privacy concerns, where the effect of the approximation error has been explicitly analysed. If the input-distribution discrepancies are available, they can instead be incorporated into the input-discrepancy-based weight-design objective derived from bound (5).

Reviewer_hRVyOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies online knowledge distillation over decentralized networks, where nodes collaboratively train local models using their own labeled data and the evolving predictions of neighboring models. It makes three main contributions. First, it derives a finite-round convergence bound that controls the drift of local objectives through neighboring nodes’ gradients and stochastic noise, avoiding the uniform gradient-norm or objective-drift bounds used in related analyses. Second, it combines convergence and generalization analyses to derive personalized test-risk bounds that explicitly characterize the effects of data-distribution discrepancies, dataset sizes, the supervised-versus-distillation trade-off, and the ensemble-weight matrix. Third, it uses a surrogate of the generalization bound to optimize ensemble weights, yielding the proposed GBOW method.

The theoretical results are supported by comparisons with existing convergence analyses and numerical experiments on MNIST, FashionMNIST, and CIFAR-10, with additional experiments involving heterogeneous model architectures and sparse communication graphs. GBOW achieves the highest observed best-over-ρ average personalized test accuracy in 14 of 15 reported dataset and label-partition settings. Overall, the paper provides a theoretically motivated framework for understanding and optimizing decentralized online distillation, although the practical scope of the results depends on the validity of the assumptions and the extent to which the experiments demonstrate generalization beyond the evaluated settings.

Critical strengths
  1. A more informative finite-round convergence analysis. The paper controls the drift of time-varying distillation objectives through neighboring models’ gradients and stochastic noise, rather than relying on uniform bounds on gradient norms or objective drift. This provides a more adaptive analysis of inter-node interactions and establishes conditions under which the resulting convergence bound is tighter than related alternatives.

  2. A useful connection between theory and algorithm design. The generalization analysis explicitly characterizes how distribution mismatch, sample sizes, supervision strength, and ensemble weights influence personalized test risk through the network. The resulting GBOW method translates this analysis into an actionable ensemble-weight optimization procedure, with experiments showing consistent improvements over several teacher-design baselines.

Critical weaknesses
  1. Restrictive assumptions behind the generalization guarantees. The generalization analysis relies on a Polyak–Łojasiewicz condition, bounded minimum empirical composite losses, and a shared conditional label distribution across nodes and the reference dataset. These assumptions may limit applicability to realistic heterogeneous learning environments. The paper would benefit from a clearer discussion of how often these conditions are expected to hold and how sensitive the guarantees are when they are violated.

  2. Limited empirical evidence for the practical generality of the proposed method. The experiments primarily use MNIST, FashionMNIST, and CIFAR-10 with 20 nodes and controlled data partitions. Although additional experiments examine architectural heterogeneity and communication sparsity, the evaluation does not establish performance on larger-scale or more realistic decentralized tasks. Moreover, the reported best-over-ρ results do not by themselves demonstrate that the method can select all relevant hyperparameters effectively without access to test performance. These limitations weaken the evidence for broad practical applicability, although they do not invalidate the theoretical contributions.

Questions for authors
  1. Clarification of assumptions: How realistic is the shared conditional label-distribution assumption in the intended applications? Can the generalization guarantee be extended, or its degradation quantified, when nodes exhibit conditional distribution shift rather than only differences in their input marginals?

  2. Practical hyperparameter selection: In the experiments, each method is evaluated using its own best-performing value of ρ. Can the authors demonstrate that GBOW provides improvements when the trade-off parameter is selected using only training data or a validation set, without using test accuracy for model selection?

  3. Robustness of the bound-guided optimization: How sensitive are the learned ensemble weights to errors in estimating label-marginal discrepancies, finite-sample variability, and the choice of optimization initialization? Do the theoretical error terms derived in the appendix adequately predict the empirical sensitivity?

  4. Scope of the convergence improvement: Can the authors provide representative numerical comparisons of the actual convergence behavior and the tightness of the competing theoretical bounds, beyond identifying parameter regimes where the proposed bound is provably smaller?

1 comment

We thank the reviewer for their positive evaluation and constructive comments. The following are our response to the questions raised.

W1 and Q1: Clarification of assumptions.

  1. Role of the PL condition. The generalization bound can be established without the PL condition or Assumption 10 when expressed in terms of the empirical composite losses achieved by the output models. Specifically, retaining the empirical-loss terms before the PL substitution in Eq. (58), and following the remaining proof steps, gives $\boldsymbol R \preceq{} (I-qW)^{-1} \left( \widehat{\boldsymbol f}+2\rho\boldsymbol{\Re} \right)+(I-q^2W^2)^{-1}\boldsymbol B+\frac{q+r}{\rho}\mathbf1$, where f^:=(E[fi,θ^(θ^i)])i=1N\hat{\boldsymbol{f}} := \left(\mathbb{E}\left[f_{i,\hat{\boldsymbol{\theta}}}(\widehat{\theta}_{i})\right]\right)_{i=1}^{N}. This generalization bound remains valid without the PL condition and Assumption 10. These two assumptions are only used to replace these achieved empirical losses by $\mathbb E f_{i,\widehat{\boldsymbol\theta}}(\widehat\theta_i) \leq \epsilon_i+\frac{s_i}{2\mu_i},$ which connects the generalization bound to our convergence analysis and yields the explicit finite-round certificate in Theorem 5. Note that the PL condition permits non-convex functions, which can be more general than convexity.

  2. Bounded minimum empirical composite loss. Assumption 10 bounds the infimum of the composite loss, which can be easily satisfied. For example, if the hypothesis class contains a uniform predictor u=(1/M,…,1/M)u=(1/M,\ldots,1/M), supervised KL divergence satisfies ℓ(u,ey)=log⁡M\ell(u,e_y)=\log M, and distillation KL satisfies DKL(t∣u)=log⁡M−H(t)≤log⁡MD_{\mathrm{KL}}(t|u)=\log M-H(t)\leq\log M, for any teacher distribution tt. An analogous finite bound holds for the squared Euclidean loss function.

  3. Shared conditional label distribution. Theorem 5 does not require any assumption on the conditional label distribution of the reference set. We have removed this unnecessary restriction in the revised manuscript. Secondly, we have added a generalization bound without the shared conditional label distribution assumption across the nodes, as shown in Appendix C.11. An explicit penalty q(I−qW)−1Cq(I-qW)^{-1}\boldsymbol C quantifies how conditional label distribution discrepancies affects the guarantee through the ensemble weights and network interactions. This bound recovers the original bound when the discrepancies vanish. Thirdly, a common PY∣X\mathcal{P}_{Y\mid X} describes the feature-distribution-skew setting, which is commonly considered in distributed learning literature [1]-[3]. It is appropriate, for example, when there is an underlying shared labeling function across nodes.

References

[1][1] Yan, Yunlu, et al. "A simple data augmentation for feature distribution skewed federated learning." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

[2][2] Kairouz, Peter, and H. Brendan McMahan. "Advances and open problems in federated learning." Foundations and trends in machine learning 14.1-2 (2021): 1-210.

[3][3] Hu, Ke, et al. "Feature Norm Regularized Federated Learning: Utilizing Data Disparities for Model Performance Gains." IJCAI. 2024.

W2 and Q2: Empirical generality and hyperparameter selection. We have added a 50-node CIFAR-10 experiment in Appendix D.9 with validation-based selection of ρ\rho. The results show that GBOW retains its performance advantage in the larger 50-node network. Specifically, GBOW achieves the highest observed mean test accuracy both when each method selects ρ\rho using validation accuracy (Table 19) and at every evaluated fixed value of ρ\rho (Table 20).

Q3: Robustness of the bound-guided optimization. We have added a sensitivity analysis in Appendix D.10, examining perturbations of the label-marginal inputs and different optimization initializations. We report changes in the returned weight matrix and the achieved surrogate objective. Specifically, we perturb the class proportions supplied to GBOW to assess how inaccuracies in these inputs affect the optimization results. Such inaccuracies can arise from finite-sample estimation. We thus do not separately model the estimation error.

Appendix C.10 (originally C.9) quantifies the change in the right-hand side of the generalization inequality when input-distribution discrepancies are replaced by empirical label-marginal discrepancies, at the same weight matrix WW. The related error terms do not directly bound changes in the optimized weights resulting from changes in the GBOW inputs or initialization.

Q4: Scope of the convergence improvement. We have added a numerical example in Appendix D.8, verifying the assumptions with explicit constants and deriving closed-form expressions for all three convergence bounds. We compare these bounds with the actual stationarity quantity S0S_0 based on an actual training trajectory. The numerical results quantify the improvements of the bound in Theorem 1 over the bounds in Theorems 2 and 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.