- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies aggregation under partial model reception in decentralized federated learning. It compares a realized-coordinate average, MFG, with imputation and loss-compensation rules, analyzing per-step convexity and inverse-probability variance. Experiments on 20 clients examine how constant versus cosine learning-rate schedules affect accuracy, aggregation fidelity, consensus, and rounds-to-target. The main datasets are CIFAR-10 and EMNIST, with a CIFAR-100/ResNet-18 extension. The central claim is that annealing can conceal aggregation differences in final accuracy.
Critical strengths
-
The evaluation question is useful: final accuracy can conceal differences in training trajectories and the cost of reaching a target. Comparing two schedules with fidelity, consensus, and rounds-to-target provides a more informative assessment than endpoint accuracy alone.
-
Proposition 5.3 correctly establishes coordinate-diameter non-expansion for convex aggregation in each realization. Together with the explicit expected mixing coefficient in Proposition 5.4, this explains the rule's self-retention behavior as reception decreases.
-
The reported design pairs data partitions, initializations, and reception masks across methods, and uses GroupNorm to avoid averaging running statistics. Learning-rate and topology ablations, plus the larger ResNet-18 model, address several plausible confounds within the tested setting.
Critical weaknesses
-
Dataset and network scale are limited. Evaluation uses relatively small, low-resolution benchmarks: CIFAR-10/100 (32x32) and EMNIST (28x28), all with only 20 clients (Appendix C.1). The CIFAR-100/ResNet-18 extension increases model size but provides limited evidence beyond small-image tasks. Please add a larger-scale benchmark such as ImageNet-1K with higher-resolution inputs to test whether the accuracy-fidelity separation and convergence-cost ordering persist on a more demanding task. Also extend the client count to at least 50, preferably 100, with comparable average degree and per-client work. Applicability to larger datasets and decentralized networks therefore remains untested.
-
The accuracy-equivalence claim is too broad. Under cosine annealing at 90% loss, Table 1 reports EMNIST accuracy of 83.9+/-0.1 for MFG versus 81.9+/-0.3 for Soft-DSGD, so final accuracy still separates these methods. Conversely, overlapping standard deviations on CIFAR-10 do not establish equivalence. With only three seeds and no stated equivalence margin or paired confidence intervals, the headline conclusion needs narrower scope and a suitable statistical analysis.
-
Fidelity comparisons mix operator effects with reference and trajectory differences. Section 7.2 measures distance to an unweighted neighbor mean, while the lossless D-PSGD reference uses Metropolis weights (Appendix C.5). These targets differ on irregular graphs. Raw Q(t) also depends on each method's current disagreement, so comparisons along separate training trajectories do not isolate the aggregation operator. A common-state replay and errors relative to each rule's own lossless operator would clarify whether the reported ratios reflect loss robustness or nominal mixing and trajectory differences.
-
Corollary 4.3 does not establish collapse during training. Appendix A.3 iterates the aggregation-only contraction while omitting ongoing SGD. For identical scalar objectives f(x)=(x-a)^2/2 on a regular graph, one SGD step followed by ZeroFill gives the expected-iterate recursion z_next=rho*((1-eta)z+etaa). For constant 0<eta<1 and 0<rho<1, its fixed point is rhoetaa/[1-rho*(1-eta)], which is nonzero for a nonzero a. Thus the stated geometric decay to zero and consequent chance accuracy require additional conditions; the observed failures alone do not prove this general claim.
-
The released implementation has a reproducibility blocker. In the Zenodo archive, node.py calls the aggregator with four positional arguments, but the registered DFL-AA implementation, IPWGossip.aggregate in aggregators.py, accepts only local and received. Static inspection therefore identifies an interface mismatch in the provided experiment path. Please release the runnable version and exact configurations that generated the reported DFL-AA results; these comparisons are central to the claimed within-family ordering.
Questions for authors
-
Do the main rankings persist with 50 and 100 clients at comparable average degree and per-client work? Can one larger-scale setting also test heterogeneous or temporally correlated reception?
-
Which dataset/loss settings support accuracy equivalence under a prespecified practical margin, and how do you reconcile that claim with the EMNIST differences in Table 1?
-
Does the fidelity ordering persist when all aggregators are evaluated on identical model states and masks, using both a common target and their respective lossless operators?
-
Can Corollary 4.3 be restated for aggregation alone, or extended to include gradient forcing and identify conditions under which training actually collapses?
-
Which runnable source version generated the DFL-AA results, and how is the aggregator-call signature mismatch resolved?
2 comments
Thank you for the detailed response and revisions. The common-state replay clarifies the fidelity comparison, and the revised Corollary 4.3 addresses my concern about ZeroFill. I am maintaining my original rating because the empirical evaluation remains limited to 20-client networks with homogeneous, independent packet loss. It therefore remains unclear whether the rankings in aggregation fidelity and rounds to target would hold in larger networks or under heterogeneous or temporally correlated packet loss.
Thank you for a very thorough review, including checking our code and working out a counterexample. Each point led to a concrete change, shown in blue in the revised PDF.
Corollary 4.3 (W4, Q4). You are right. The old proof (now App. B.3) iterated the aggregation step alone and left out the gradient steps. The corrected corollary includes them and says when collapse actually happens. (a) For f(x)=21(x−a)2 with k gradient steps per round, the expected iterate settles at sa with s=ρ(1−β)/(1−ρβ)<1 and β=(1−η)k. With k=1 this is exactly your fixed point. So at a constant step size, zero-fill shrinks a single-layer model rather than wiping it out, and s→0 as the step size goes to zero. (b) For a product of L weights, f(w)=21(∏lwl−a)2, the all-zero model is a fixed point. With two layers it becomes stable once loss passes a threshold, and with three or more layers it is stable at every loss rate, because the gradient that would restore one layer is proportional to the other layers' weights. In our setting, (a) gives s=0.50 at 10% loss and 0.03 at 90%, the two-layer threshold is 10.6% loss, and a simulation matches these values to three decimals. We state clearly that local stability alone does not prove training falls into this state. That it does, at every loss level we tested, is shown in Table 1.
Fidelity comparisons (W3, Q3). We agree with both concerns and ran the experiment you suggested (new App. D.3). We saved node states from two lossless D-PSGD runs, one with a constant and one with a cosine step size, so the states belong to none of the compared rules. Every rule then received the same states and the same loss patterns, and we measured its error both against the plain neighborhood average and against its own output with nothing lost. MFG has the lowest error of every rule at every loss level, measured either way, so its lead comes from the rule itself and not from its training path. DFL-AA's output with nothing lost equals the plain average, so that comparison does not depend on the reference, and MFG's error is 1.4 to 2.4 times lower. The larger gaps in Section 7.2 also include the extra disagreement DFL-AA builds up during training. Your concern about the reference is correct for Soft-DSGD. Against the plain average it looks 1.4 to 4.5 times worse than MFG, but against its own weighted output only 1.1 to 2.1 times, which puts it closer to MFG than DFL-AA. We now claim only MFG's lead at this level. The cosine replay gives the same ratios for the stable rules, while the unnormalized rules' errors become 107 to 108 times MFG's, because annealing brings nodes together but does not shrink the parameters.
Accuracy equivalence (W2, Q2). We agree, and the abstract, Section 7.2 and Section 8 no longer claim equivalence. Your EMNIST example is correct. App. D.4 now gives, for every cosine cell, a 95% interval for the accuracy difference and an equivalence test with a ±1 point margin chosen in advance. On EMNIST, MFG is within ±1 point of both baselines at 10 to 20% loss and significantly better at 50 to 90% loss. On CIFAR-10 every cell is inconclusive, since three seeds can show neither a difference nor equivalence, and we now say exactly that.
Code (W5, Q5). You are right. While cleaning the code for release, we removed two unused arguments from the DFL-AA aggregator, which broke the call in node.py. The experiments ran with the original version, and DFL-AA does not use those arguments, so the results are unaffected. The new archive (https://zenodo.org/records/24299491) restores the original signature.
Scale (W1, Q1). We agree this is the main limitation of the current evaluation, and Section 8 now says so. Runs with 50 and 100 nodes and on Tiny-ImageNet are planned for the extended version.
Thank you again. Your corrections made the paper more accurate. We are happy to answer any further questions.