- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies online knowledge distillation over decentralized networks of heterogeneous models and personalized data distributions. Unlike offline distillation with a fixed pretrained teacher, each node’s teacher here is a weighted average of its neighbors’ predicted probabilities, using fixed weights and an optional self-weight, evaluated at the round-start parameters and refreshed every communication round. The teachers therefore evolve together with the students, coupling the per-node distillation objectives and making them time-varying. The authors claim three main contributions. First, they propose a finite-round convergence bound (Theorem 1) that controls objective drift via neighbors’ local gradients and stochastic noise, avoiding the uniform bounds on expected stochastic-gradient norms used by Tan et al. (2022) and on round-to-round objective drift used by Abourayya et al. (2025a). They provide a counterexample (Appendix C.5) intended to show that their assumptions do not imply the uniform gradient bound, and necessary/sufficient conditions (Corollary 1 and Theorem 4) under which their bound is tighter than existing bounds. Second, they combine this convergence analysis with a generalization analysis to bound the expected personalized test risk of the finite-round output models (Theorem 5), with explicit dependence on distribution discrepancies, dataset sizes, the trade-off factor, and the ensemble weight matrix, including how these effects propagate through the network. Third, they propose GBOW, an ensemble-weight optimization method derived from a surrogate of this bound. The paper reports experiments on MNIST, FashionMNIST, and CIFAR-10 across five heterogeneity levels (20 nodes; complete graph by default), where GBOW attains the highest best-over-ρ accuracy in 14 of 15 settings against seven weight-design baselines, along with additional experiments on heterogeneous architectures and sparse communication graphs. Proofs are provided in Appendix C, including an approximation and estimation error analysis for the label-marginal proxies used by GBOW (Appendix C.9); experimental details are in Appendix D.
Critical strengths
1.The paper presents the first end-to-end convergence-and-generalization analysis whose bound applies to the actual finite-round training output. Unlike the closest prior analyses---which bound idealized ERM ensembles (Cho et al., 2023), teacher-dependent quantities with no explicit parameter dependence (Shao et al., 2024), or convex KRR estimators (Park et al.) Theorem 5 bounds the expected personalized test risk of the models actually produced by the coupled, time-varying online-distillation procedure, with explicit dependence on the ensemble weights and the trade-off factor. This is the strongest consideration supporting acceptance: it is the paper's genuinely novel contribution and the foundation for the bound-guided weight design (GBOW) underlying the reported empirical gains.
2.The improvement claims are rigorously qualified rather than merely asserted. The paper shows, by a constructive counterexample, that its assumptions do not imply the uniform gradient bound of Tan et al. (2022) (Appendix C.5), and characterizes the claim of a tighter bound through an exact necessary-and-sufficient condition (Lemma 8), a necessary condition (Corollary 1), a sufficient condition (Theorem 4), and a monotonicity analysis (Appendix C.4). For an improvement-oriented paper, the credibility of the improvement claim is central.
Critical weaknesses
1.As presented, the key improvement claim is established only under conditions and is not yet quantified. The conditions in Lemma 8, Corollary 1, Theorem 4, and Appendices C.4/C.6 involve constants that are not directly observable in practice, and the paper does not include an instance showing that the conditions can be met or how large the improvement is. Consequently, it is difficult for a reader to assess whether the promise of "tighter guarantees in wide-range parameter regimes" is non-vacuous in any concrete setting—the consideration that most limited my rating, though a single worked example would likely address it.
Questions for authors
Can you provide one concrete instance---e.g., a simple model where , , , , and can all be instantiated, demonstrating that , including the size of the gap as a function of the problem parameters?
2 comments
Thank you for adding the worked example in Appendix D.8. Taking the reported derivations at face value, the example does what I asked for. It instantiates the relevant constants and shows the resulting gap so my W1/Q1 is addressed and I have no further questions on it.
The example resolves the specific concern I raised but it does not alter my overall assessment of the paper's contributions. I keep my rating at 3, weak acceptance.
W1 and Q1: Example with quantified constants and bounds. We thank the reviewer for their excellent suggestion. In the revised manuscript, we have added a fully specified two-node example in Appendix D.8 to quantify the theoretical constants and the corresponding performance improvement.
In particular, we show that this example gives $ L=1,\qquad L_h=\frac{\sqrt2}{2},\qquad L_d=\sqrt2, \qquad \sigma_{\mathrm{sgd}}=C_{\mathrm{sgd}}=0, \qquad U=\rho, \text{ and } \widetilde U=8(1-\rho)\rho/27. $ We then derive closed-form expressions for all three convergence bounds and exact conditions for their comparison as functions of ρ and the number of rounds T. We also numerically evaluate the actual stationarity quantity S0 and report it alongside these bounds for T∈{10,100,1000,10000}. For example, at ρ=0.9 and T=1000, we have $ S_0\approx0.000781,\qquad Q\approx0.04246,\qquad \widetilde Q\approx0.18125,\text{ and } Q_3\approx0.05458. $ Thus, our bound in Theorem 1 is 76.57% smaller than the bound in Theorem 2 and 22.21% smaller than that in Theorem 3. Moreover, the derived conditions give Q<Q for every integer T≥2 and Q<Q3 for every integer T≥71.