- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes trajectory consistency-based stopping, which adaptively selects each client number of local SGD steps by comparing the direction and magnitude of its update against a early round reference. The method combines this criterion with normalized aggregation and provides theoretical analysis and experiments under a fixed local update budget.
Critical strengths
The stopping criterion uses quantities already available during local SGD, making it inexpensive to compute without additional model evaluations or communication during local training.
Critical weaknesses
- The controlled heterogeneity study uses only MNIST with Dirichlet constants alpha = 0.1 and 10, covering strongly heterogeneous and approximately IID settings without examining intermediate regimes. What's worse is that the main CIFAR-100 experiment uses a single pathological partition. This provides limited evidence that the proposed rule consistently handles different degrees of heterogeneity. A broader sweep on CIFAR-100, together with a precise description of the partition and resulting client distributions is needed.
- CIFAR-100 accuracy remains around 25% despite using ResNet-18. These results do not establish whether TCS improves eventual performance or primarily benefits the early optimization trajectory where ResNet18 can easily achieve post-40% accuracy. Relative rankings can change with longer training, and methods that initially perform worse may ultimately reach higher accuracy. The authors should verify the implementation and training configuration, report longer-budget learning curves, and establish whether the gains persist when the methods reach substantially higher accuracy or approach convergence.
- Table 2 matches local optimizer-update counts, while communication rounds differ substantially. For example, TCS uses approximately 772 rounds on CIFAR-100, compared with approximately 101 for MILLS. Its higher accuracy therefore comes with much more frequent synchronization, which is itself a potential explanation for the improvement. A fair communication budgeted evaluation under matched network-exchange budgets is needed.
Questions for authors
Please address the concerns above, particularly the low CIFAR-100 accuracy. The current results make it difficult to determine whether the reported gains persist beyond early training. Longer-budget experiments with well-tuned baselines approaching convergence are needed to substantiate the method’s effectiveness.
4 comments
Response to Reviewer V998 (2/2)
For convenience, we provide the complete result tables below.
Heterogeneity comparisons
MNIST
(each cell: average accuracy ± standard deviation / average NLL / average communication rounds)
| Method | α=0.01 | α=0.1 | α=1 |
|---|---|---|---|
| Fixed-10 | 95.12 ± 0.37 / 0.152 / 1,000.0 | 96.45 ± 0.37 / 0.114 / 1,000.0 | 97.51 ± 0.14 / 0.078 / 1,000.0 |
| Fixed-20 | 86.70 ± 4.86 / 0.410 / 500.0 | 95.29 ± 0.45 / 0.144 / 500.0 | 97.26 ± 0.05 / 0.087 / 500.0 |
| Fixed-50 | 74.70 ± 6.45 / 0.786 / 200.0 | 92.70 ± 0.59 / 0.229 / 200.0 | 96.41 ± 0.05 / 0.115 / 200.0 |
| Fixed-100 | 41.00 ± 16.13 / 2.691 / 100.0 | 92.44 ± 0.45 / 0.249 / 100.0 | 95.42 ± 0.11 / 0.151 / 100.0 |
| Mills | 90.87 ± 0.30 / 0.273 / 530.0 | 95.67 ± 0.20 / 0.137 / 530.0 | 97.25 ± 0.09 / 0.086 / 530.0 |
| AdaComm | 68.54 ± 9.64 / 1.122 / 114.0 | 90.75 ± 5.00 / 0.302 / 186.3 | 97.03 ± 0.07 / 0.096 / 311.7 |
| QSR | 91.38 ± 3.45 / 0.268 / 906.0 | 96.43 ± 0.25 / 0.111 / 906.0 | 97.50 ± 0.10 / 0.079 / 906.0 |
| ALT | 71.30 ± 11.41 / 1.047 / 234.0 | 91.78 ± 0.47 / 0.260 / 160.0 | 95.69 ± 0.15 / 0.144 / 104.0 |
| FLASH | 89.96 ± 1.02 / 0.315 / 436.0 | 96.29 ± 0.10 / 0.116 / 892.0 | 97.58 ± 0.08 / 0.079 / 950.0 |
| TCS κ=0.1 | 88.17 ± 2.15 / 0.373 / 377.7 | 95.35 ± 0.37 / 0.145 / 453.3 | 97.30 ± 0.21 / 0.085 / 512.3 |
| TCS κ=0.3 | 89.68 ± 2.53 / 0.326 / 451.7 | 95.98 ± 0.35 / 0.129 / 566.7 | 97.47 ± 0.16 / 0.082 / 642.3 |
| TCS κ=0.5 | 91.79 ± 1.22 / 0.252 / 537.7 | 95.84 ± 0.66 / 0.131 / 681.0 | 97.44 ± 0.13 / 0.082 / 747.7 |
| TCS κ=0.7 | 91.76 ± 2.13 / 0.252 / 649.0 | 96.46 ± 0.15 / 0.113 / 799.0 | 97.54 ± 0.09 / 0.080 / 841.3 |
| TCS κ=0.9 | 93.63 ± 0.41 / 0.201 / 794.7 | 96.39 ± 0.22 / 0.113 / 899.7 | 97.49 ± 0.15 / 0.077 / 908.0 |
CIFAR-100
(each cell: average accuracy (standard deviation) / average communication rounds)
| Method | CIFAR-100/20 | CIFAR-100/50 | CIFAR-100/100 |
|---|---|---|---|
| Fixed-10 | 38.02 (1.22) / 3,000 | 41.68 (0.74) / 3,000 | 43.72 (0.55) / 3,000 |
| Fixed-20 | 33.68 (0.37) / 1,500 | 38.66 (0.79) / 1,500 | 42.17 (0.83) / 1,500 |
| Fixed-50 | 26.14 (0.93) / 600 | 30.25 (1.02) / 600 | 35.02 (0.98) / 600 |
| Fixed-100 | 20.59 (1.02) / 300 | 25.19 (0.46) / 300 | 29.72 (0.42) / 300 |
| Mills | 19.62 (1.26) / 329 | 24.90 (0.70) / 345 | 28.89 (0.36) / 358 |
| AdaComm | 19.98 (0.24) / 312 | 24.70 (0.26) / 320 | 28.98 (0.41) / 342 |
| QSR | 37.14 (0.97) / 2,775 | 40.20 (0.85) / 2,775 | 43.45 (0.17) / 2,775 |
| ALT | 28.35 (2.44) / 1,101 | 34.02 (0.73) / 865 | 35.57 (0.79) / 638 |
| FLASH | 30.33 (0.86) / 2,747 | 39.31 (0.66) / 2,768 | 43.19 (0.54) / 2,791 |
| TCS κ=0.1 | 33.47 (0.52) / 1,570 | 36.55 (0.94) / 1,546 | 41.21 (0.21) / 1,501 |
| TCS κ=0.3 | 37.35 (0.38) / 1,924 | 37.45 (0.78) / 1,898 | 42.36 (0.64) / 1,766 |
| TCS κ=0.5 | 36.53 (1.01) / 2,280 | 39.61 (0.62) / 2,264 | 43.53 (0.68) / 2,272 |
| TCS κ=0.7 | 37.90 (0.41) / 2,598 | 41.61 (0.82) 2,599 | 41.75 (0.32) / 2,600 |
| TCS κ=0.9 | 37.26 (0.36) / 2,727 | 40.06 (0.35) / 2,727 | 41.74 (0.31) / 2,727 |
Comparison under fixed-communication setting (1000 rounds)
(each cell: final accuracy / cumulative SGD works (x1000))
| Method | FEMNIST | CIFAR-100/20 | CIFAR-100/50 | CIFAR-100/100 | StackOverflow |
|---|---|---|---|---|---|
| Fixed-10 | 68.49 / 20.0 | 23.09 / 50.0 | 34.09 / 50.0 | 41.27 / 50.0 | 18.40 / 100.0 |
| Fixed-20 | 75.06 / 40.0 | 31.41 / 100.0 | 39.68 / 100.0 | 42.54 / 100.0 | 19.61 / 200.0 |
| Fixed-50 | 76.86 / 100.0 | 28.62 / 250.0 | 28.20 / 250.0 | 30.93 / 250.0 | 20.48 / 500.0 |
| Fixed-100 | 76.10 / 200.0 | 24.26 / 500.0 | 23.65 / 500.0 | 25.95 / 500.0 | 19.82 / 1,000.0 |
| Mills | 76.46 / 131.4 | 24.02 / 366.3 | 23.43 / 338.3 | 26.16 / 328.7 | 20.59 / 795.9 |
| AdaComm | 76.30 / 136.2 | 24.41 / 384.1 | 23.91 / 338.6 | 26.92 / 320.9 | 20.33 / 854.8 |
| QSR | 67.17 / 22.4 | 24.07 / 56.0 | 33.03 / 56.0 | 41.73 / 56.0 | 18.44 / 112.0 |
| ALT | 76.19 / 199.9 | 29.59 / 134.7 | 39.25 / 164.0 | 41.39 / 193.7 | 19.99 / 972.8 |
| FLASH | 68.21 / 21.2 | 23.16 / 53.4 | 34.38 / 52.6 | 41.02 / 52.3 | 18.48 / 105.9 |
| TCS κ=0.1 | 76.78 / 46.1 | 31.09 / 97.3 | 39.95 / 98.6 | 42.16 / 100.1 | 20.19 / 348.9 |
| TCS κ=0.3 | 76.51 / 33.8 | 29.03 / 78.7 | 40.12 / 79.5 | 42.08 / 80.2 | 19.43 / 230.1 |
| TCS κ=0.5 | 71.70 / 26.6 | 27.57 / 65.0 | 37.99 / 65.1 | 41.13 / 64.4 | 19.97 / 159.8 |
| TCS κ=0.7 | 69.89 / 23.0 | 26.06 / 56.8 | 34.93 / 56.3 | 41.18 / 55.8 | 19.66 / 121.0 |
| TCS κ=0.9 | 69.30 / 22.0 | 25.66 / 55.0 | 35.39 / 55.0 | 41.02 / 55.0 | 19.63 / 110.0 |
Thank you for the substantial additional experiments. The expanded partitions largely address my heterogeneity concern, and the longer-budget and equal-round results provide useful evidence of favorable trade-offs in some settings. My prior concern regarding the training accuracy being suspiciously low has been greatly addressed and I think the new results greatly supported the claims the papers are making. I will therefore raise my score to 3.
We are tremendously grateful for your thoughtful reassessment. Your constructive feedback indeed helped us strengthen the experiments and improve the manuscript. Thank you for your time and careful consideration.
Response to Reviewer V998 (1/2)
We thank the reviewer for identifying limitations in heterogeneity coverage, CIFAR-100 training duration, and the interpretation of communication frequency. We have expanded the heterogeneity study, tripled the main CIFAR-100 update budget, and added a separate fixed-communication comparison reported in this response.
1. Broader and precisely defined heterogeneity (W1)
The revised MNIST comparison uses Dirichlet α∈{0.01,0.1,1} with three paired seeds, covering three levels of label heterogeneity (Section 5.3, lines 465–471; Table 5).
For CIFAR-100, we evaluate 20, 50, and 100 allocated classes per client (i.e., pathological non-IID setting), with five paired seeds per condition. All settings retain 500 clients, 100 training images per client, and the same 100-class prediction task. Each client receives 5, 2, or 1 image per assigned class, respectively. This varies client label support while keeping dataset size and the classification task fixed.
The partition and training configurations are described in Section 5.1 and Appendix B.2 (lines 371–374, 412–413, and 1009–1012; Table 3). Tables 2 and 5 give the complete CIFAR-100 and MNIST results. We also reproduce these tables in part 2/2 for convenience.
2. Longer training and stronger CIFAR-100 results (W2)
We agree that the original 100k-update budget, which yields approximately 25% CIFAR-100 accuracy, was insufficient to assess performance beyond early training. We increased the main CIFAR-100 budget from 100k to 300k local updates and report five-seed endpoint results. The model is ResNet-18 trained from scratch with a CIFAR-size stem and group normalization, using batch size 10, SGD without momentum, and initial learning rate 0.1. The configuration is documented in Table 3 and Appendix B.1–B.2 (lines 990–999 and 1009–1012).
At the same κ=0.7 across all three partitions, Table 2 reports:
At this common threshold, the means are close on the 20- and 50-class partitions, while the 100-class partition retains a 1.97-point gap. On the latter partition, κ=0.5 gives a more favorable operating point: 43.53 ± 0.68% in 2,272 rounds versus Fixed-10's 43.72 ± 0.55% in 3,000 rounds, reducing rounds by 24.3%.
Section 5.2 discusses these differences across partitions and thresholds (lines 426–431; 454-459). The extended endpoints show that favorable accuracy–communication trade-offs remain available at a substantially longer training horizon. We report these as fixed-work results, without claiming that every method has reached convergence.
3. Communication-budgeted evaluation (W3)
We increased the main CIFAR-100 budget from 100k to 300k local updates and report five-seed endpoint results. The model is ResNet-18 trained from scratch with a CIFAR-size stem and group normalization, using batch size 10, SGD without momentum, and initial learning rate 0.1. The configuration is documented in Table 3 and Appendix B.1–B.2 (lines 990–999 and 1009–1012).
The new five-seed results provide complementary evidence:
While these do not imply universal superiority, the two protocols importantly answer complementary questions:
Because the protocols differ in their resource constraints and training schedules, we interpret each comparison within its own setting rather than comparing absolute accuracies across protocols. We believe these results support TCS as a low-overhead mechanism for adapting client-specific local computation, with benefits that depend on the dataset and stopping threshold.
Once again, we would greatly appreciate your input, which enriched our manuscripts and presentation of results.
References