acceptodds
Under review as a conference paper at ICLR 2027

Halt before Deviate! Adaptive Local Effort in Federated Learning

Abstract

The number of local steps is an important control knob in federated learning, shaping both communication cost and the optimization trajectory. However, heterogeneous data can cause local updates to drift and hinder global progress. Choosing when to stop local training is therefore essential to balance communication efficiency and optimization progress. We propose an adaptive stopping rule that selects each client's local step count by comparing both the direction and magnitude of its average update with a reference formed early in the round. Our analysis explains how the rule responds to differences in local curvature and relates local alignment to global descent. Under a common budget of local updates, the proposed rule achieves comparable accuracy with fewer communication rounds.

Primary area:
optimization
Keywords:
federated learning, adaptive local steps, distributed optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_V998OpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes trajectory consistency-based stopping, which adaptively selects each client number of local SGD steps by comparing the direction and magnitude of its update against a early round reference. The method combines this criterion with normalized aggregation and provides theoretical analysis and experiments under a fixed local update budget.

Critical strengths

The stopping criterion uses quantities already available during local SGD, making it inexpensive to compute without additional model evaluations or communication during local training.

Critical weaknesses
  1. The controlled heterogeneity study uses only MNIST with Dirichlet constants alpha = 0.1 and 10, covering strongly heterogeneous and approximately IID settings without examining intermediate regimes. What's worse is that the main CIFAR-100 experiment uses a single pathological partition. This provides limited evidence that the proposed rule consistently handles different degrees of heterogeneity. A broader sweep on CIFAR-100, together with a precise description of the partition and resulting client distributions is needed.
  2. CIFAR-100 accuracy remains around 25% despite using ResNet-18. These results do not establish whether TCS improves eventual performance or primarily benefits the early optimization trajectory where ResNet18 can easily achieve post-40% accuracy. Relative rankings can change with longer training, and methods that initially perform worse may ultimately reach higher accuracy. The authors should verify the implementation and training configuration, report longer-budget learning curves, and establish whether the gains persist when the methods reach substantially higher accuracy or approach convergence.
  3. Table 2 matches local optimizer-update counts, while communication rounds differ substantially. For example, TCS uses approximately 772 rounds on CIFAR-100, compared with approximately 101 for MILLS. Its higher accuracy therefore comes with much more frequent synchronization, which is itself a potential explanation for the improvement. A fair communication budgeted evaluation under matched network-exchange budgets is needed.
Questions for authors

Please address the concerns above, particularly the low CIFAR-100 accuracy. The current results make it difficult to determine whether the reported gains persist beyond early training. Longer-budget experiments with well-tuned baselines approaching convergence are needed to substantiate the method’s effectiveness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer V998 (1/2)

We thank the reviewer for identifying limitations in heterogeneity coverage, CIFAR-100 training duration, and the interpretation of communication frequency. We have expanded the heterogeneity study, tripled the main CIFAR-100 update budget, and added a separate fixed-communication comparison reported in this response.

1. Broader and precisely defined heterogeneity (W1)
The revised MNIST comparison uses Dirichlet α∈{0.01,0.1,1}\alpha\in\{0.01,0.1,1\} with three paired seeds, covering three levels of label heterogeneity (Section 5.3, lines 465–471; Table 5).

For CIFAR-100, we evaluate 20, 50, and 100 allocated classes per client (i.e., pathological non-IID setting), with five paired seeds per condition. All settings retain 500 clients, 100 training images per client, and the same 100-class prediction task. Each client receives 5, 2, or 1 image per assigned class, respectively. This varies client label support while keeping dataset size and the classification task fixed.

The partition and training configurations are described in Section 5.1 and Appendix B.2 (lines 371–374, 412–413, and 1009–1012; Table 3). Tables 2 and 5 give the complete CIFAR-100 and MNIST results. We also reproduce these tables in part 2/2 for convenience.

2. Longer training and stronger CIFAR-100 results (W2)
We agree that the original 100k-update budget, which yields approximately 25% CIFAR-100 accuracy, was insufficient to assess performance beyond early training. We increased the main CIFAR-100 budget from 100k to 300k local updates and report five-seed endpoint results. The model is ResNet-18 trained from scratch with a CIFAR-size stem and group normalization, using batch size 10, SGD without momentum, and initial learning rate 0.1. The configuration is documented in Table 3 and Appendix B.1–B.2 (lines 990–999 and 1009–1012).

At the same κ=0.7\kappa=0.7 across all three partitions, Table 2 reports:

Classes/clientFixed-10 accuracyTCS accuracyRound reduction
2038.02 ± 1.22%37.90 ± 0.41%13.4%
5041.68 ± 0.74%41.61 ± 0.82%13.4%
10043.72 ± 0.55%41.75 ± 0.32%13.3%

At this common threshold, the means are close on the 20- and 50-class partitions, while the 100-class partition retains a 1.97-point gap. On the latter partition, κ=0.5\kappa=0.5 gives a more favorable operating point: 43.53 ± 0.68% in 2,272 rounds versus Fixed-10's 43.72 ± 0.55% in 3,000 rounds, reducing rounds by 24.3%.

Section 5.2 discusses these differences across partitions and thresholds (lines 426–431; 454-459). The extended endpoints show that favorable accuracy–communication trade-offs remain available at a substantially longer training horizon. We report these as fixed-work results, without claiming that every method has reached convergence.

3. Communication-budgeted evaluation (W3)
We increased the main CIFAR-100 budget from 100k to 300k local updates and report five-seed endpoint results. The model is ResNet-18 trained from scratch with a CIFAR-size stem and group normalization, using batch size 10, SGD without momentum, and initial learning rate 0.1. The configuration is documented in Table 3 and Appendix B.1–B.2 (lines 990–999 and 1009–1012).

The new five-seed results provide complementary evidence:

  • CIFAR-100/50: TCS (κ = 0.3) achieves 40.12% accuracy with 79.5k updates, compared with Fixed-20's 39.68% with 100k updates. At equal rounds, TCS achieves 0.44 points higher accuracy using 20.5% less local computation.
  • CIFAR-100/100: TCS (κ = 0.3) achieves 42.08% with 80.2k updates, compared with Fixed-20's 42.54% with 100k updates, retaining near-comparable accuracy with 19.8% less computation.
  • FEMNIST: TCS (κ = 0.1) achieves 76.78% with 46.1k updates, compared with Fixed-50's 76.86% with 100k updates, using 53.9% less computation for a 0.08-point accuracy difference.

While these do not imply universal superiority, the two protocols importantly answer complementary questions:

  • Fixed work [1-3]: How much accuracy is achieved for a given computation budget, and at what communication cost?
  • Fixed communication: How much accuracy is achieved with equal synchronization opportunities, and how much local computation is required?

Because the protocols differ in their resource constraints and training schedules, we interpret each comparison within its own setting rather than comparing absolute accuracies across protocols. We believe these results support TCS as a low-overhead mechanism for adapting client-specific local computation, with benefits that depend on the dataset and stopping threshold.

Once again, we would greatly appreciate your input, which enriched our manuscripts and presentation of results.

References

    1. Don't Use Large Mini-Batches, Use Local SGD (ICLR 2020)
    1. A Quadratic Synchronization Rule for Distributed Deep Learning (ICLR 2024)
    1. The Role of Local Steps in Local SGD (Optimization Methods and Software 2025)
AuthorsOpenReview ↗

Response to Reviewer V998 (2/2)

For convenience, we provide the complete result tables below.

Heterogeneity comparisons

MNIST
(each cell: average accuracy ± standard deviation / average NLL / average communication rounds)

Methodα=0.01α=0.1α=1
Fixed-1095.12 ± 0.37 / 0.152 / 1,000.096.45 ± 0.37 / 0.114 / 1,000.097.51 ± 0.14 / 0.078 / 1,000.0
Fixed-2086.70 ± 4.86 / 0.410 / 500.095.29 ± 0.45 / 0.144 / 500.097.26 ± 0.05 / 0.087 / 500.0
Fixed-5074.70 ± 6.45 / 0.786 / 200.092.70 ± 0.59 / 0.229 / 200.096.41 ± 0.05 / 0.115 / 200.0
Fixed-10041.00 ± 16.13 / 2.691 / 100.092.44 ± 0.45 / 0.249 / 100.095.42 ± 0.11 / 0.151 / 100.0
Mills90.87 ± 0.30 / 0.273 / 530.095.67 ± 0.20 / 0.137 / 530.097.25 ± 0.09 / 0.086 / 530.0
AdaComm68.54 ± 9.64 / 1.122 / 114.090.75 ± 5.00 / 0.302 / 186.397.03 ± 0.07 / 0.096 / 311.7
QSR91.38 ± 3.45 / 0.268 / 906.096.43 ± 0.25 / 0.111 / 906.097.50 ± 0.10 / 0.079 / 906.0
ALT71.30 ± 11.41 / 1.047 / 234.091.78 ± 0.47 / 0.260 / 160.095.69 ± 0.15 / 0.144 / 104.0
FLASH89.96 ± 1.02 / 0.315 / 436.096.29 ± 0.10 / 0.116 / 892.097.58 ± 0.08 / 0.079 / 950.0
TCS κ=0.188.17 ± 2.15 / 0.373 / 377.795.35 ± 0.37 / 0.145 / 453.397.30 ± 0.21 / 0.085 / 512.3
TCS κ=0.389.68 ± 2.53 / 0.326 / 451.795.98 ± 0.35 / 0.129 / 566.797.47 ± 0.16 / 0.082 / 642.3
TCS κ=0.591.79 ± 1.22 / 0.252 / 537.795.84 ± 0.66 / 0.131 / 681.097.44 ± 0.13 / 0.082 / 747.7
TCS κ=0.791.76 ± 2.13 / 0.252 / 649.096.46 ± 0.15 / 0.113 / 799.097.54 ± 0.09 / 0.080 / 841.3
TCS κ=0.993.63 ± 0.41 / 0.201 / 794.796.39 ± 0.22 / 0.113 / 899.797.49 ± 0.15 / 0.077 / 908.0

CIFAR-100
(each cell: average accuracy (standard deviation) / average communication rounds)

MethodCIFAR-100/20CIFAR-100/50CIFAR-100/100
Fixed-1038.02 (1.22) / 3,00041.68 (0.74) / 3,00043.72 (0.55) / 3,000
Fixed-2033.68 (0.37) / 1,50038.66 (0.79) / 1,50042.17 (0.83) / 1,500
Fixed-5026.14 (0.93) / 60030.25 (1.02) / 60035.02 (0.98) / 600
Fixed-10020.59 (1.02) / 30025.19 (0.46) / 30029.72 (0.42) / 300
Mills19.62 (1.26) / 32924.90 (0.70) / 34528.89 (0.36) / 358
AdaComm19.98 (0.24) / 31224.70 (0.26) / 32028.98 (0.41) / 342
QSR37.14 (0.97) / 2,77540.20 (0.85) / 2,77543.45 (0.17) / 2,775
ALT28.35 (2.44) / 1,10134.02 (0.73) / 86535.57 (0.79) / 638
FLASH30.33 (0.86) / 2,74739.31 (0.66) / 2,76843.19 (0.54) / 2,791
TCS κ=0.133.47 (0.52) / 1,57036.55 (0.94) / 1,54641.21 (0.21) / 1,501
TCS κ=0.337.35 (0.38) / 1,92437.45 (0.78) / 1,89842.36 (0.64) / 1,766
TCS κ=0.536.53 (1.01) / 2,28039.61 (0.62) / 2,26443.53 (0.68) / 2,272
TCS κ=0.737.90 (0.41) / 2,59841.61 (0.82) 2,59941.75 (0.32) / 2,600
TCS κ=0.937.26 (0.36) / 2,72740.06 (0.35) / 2,72741.74 (0.31) / 2,727

Comparison under fixed-communication setting (1000 rounds)
(each cell: final accuracy / cumulative SGD works (x1000))

MethodFEMNISTCIFAR-100/20CIFAR-100/50CIFAR-100/100StackOverflow
Fixed-1068.49 / 20.023.09 / 50.034.09 / 50.041.27 / 50.018.40 / 100.0
Fixed-2075.06 / 40.031.41 / 100.039.68 / 100.042.54 / 100.019.61 / 200.0
Fixed-5076.86 / 100.028.62 / 250.028.20 / 250.030.93 / 250.020.48 / 500.0
Fixed-10076.10 / 200.024.26 / 500.023.65 / 500.025.95 / 500.019.82 / 1,000.0
Mills76.46 / 131.424.02 / 366.323.43 / 338.326.16 / 328.720.59 / 795.9
AdaComm76.30 / 136.224.41 / 384.123.91 / 338.626.92 / 320.920.33 / 854.8
QSR67.17 / 22.424.07 / 56.033.03 / 56.041.73 / 56.018.44 / 112.0
ALT76.19 / 199.929.59 / 134.739.25 / 164.041.39 / 193.719.99 / 972.8
FLASH68.21 / 21.223.16 / 53.434.38 / 52.641.02 / 52.318.48 / 105.9
TCS κ=0.176.78 / 46.131.09 / 97.339.95 / 98.642.16 / 100.120.19 / 348.9
TCS κ=0.376.51 / 33.829.03 / 78.740.12 / 79.542.08 / 80.219.43 / 230.1
TCS κ=0.571.70 / 26.627.57 / 65.037.99 / 65.141.13 / 64.419.97 / 159.8
TCS κ=0.769.89 / 23.026.06 / 56.834.93 / 56.341.18 / 55.819.66 / 121.0
TCS κ=0.969.30 / 22.025.66 / 55.035.39 / 55.041.02 / 55.019.63 / 110.0

Thank you for the substantial additional experiments. The expanded partitions largely address my heterogeneity concern, and the longer-budget and equal-round results provide useful evidence of favorable trade-offs in some settings. My prior concern regarding the training accuracy being suspiciously low has been greatly addressed and I think the new results greatly supported the claims the papers are making. I will therefore raise my score to 3.

We are tremendously grateful for your thoughtful reassessment. Your constructive feedback indeed helped us strengthen the experiments and improve the manuscript. Thank you for your time and careful consideration.

Reviewer_idEnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes trajectory consistency-based stopping (TCS) to choose how many local SGD steps each client takes. It compares the direction and magnitude of the average update with an early reference, then combines client updates using work-normalized aggregation. The analysis studies how curvature affects stopping and how stopping can introduce bias. Experiments compare accuracy and communication rounds under the same local-update budget.

Critical strengths
  1. TCS computes its stopping score ψi(di,h)\psi_i(d_{i,h}) from existing SGD updates. This requires no extra model evaluations or communication during local training.
  2. In the isotropic quadratic limit of Proposition 4.1, clients with higher curvature take fewer local steps HiH_i. The analysis also considers bias from stopping based on noisy updates.
  3. At κ=0.7\kappa=0.7, Table 2 reports 22.77–36.13% fewer rounds than FIXED-10 across three benchmarks, with similar or higher mean accuracy.
Critical weaknesses
  1. TCS may introduce drift even when using the same fixed step count for all clients would avoid it. Consider three quadratic clients with equal weights and the same Hessian A=diag⁡(1,2)A=\operatorname{diag}(1,2). All clients participate and use exact gradients. Their gradients at θ=0\theta=0 are (1,0),(0,1),(−1,−1)(1,0),(0,1),(-1,-1), which sum to zero. In an adaptive round with (Hmin⁡,Hmax⁡,κ,η)=(10,100,0.7,0.005)(H_{\min},H_{\max},\kappa,\eta)=(10,100,0.7,0.005), I obtain step counts (78,44,56)(78,44,56). The update in Eq. (7) then moves the model away from zero, the global optimum.
  2. Adaptive stopping may bias the combined update even when clients have the same objective and unbiased gradients. This is a general risk when stopping depends on observed gradients. Eq. (24) correctly includes βtstop\beta_t^{\mathrm{stop}}, but Theorem 4.3 does not seem to show when this bias is small for TCS.
  3. The experiments do not clearly show the benefit of checking both update direction and magnitude. Figure 2 shows which signal causes stopping, but not whether checking direction improves accuracy. Moreover, I could not find a discussion or comparison with FedVeca's client-specific step selection [1] or FLASH's validation-loss stopping [2]. A comparison would help explain what TCS adds.

[1] Luo et al. “FedVeca: Federated Vectorized Averaging on Non-IID Data With Adaptive Bi-Directional Global Objective.” IEEE Transactions on Parallel and Distributed Systems, 2024.

[2] Panchal et al. “Flash: Concept Drift Adaptation in Federated Learning.” ICML, 2023.

Questions for authors
  1. Could the authors explain when different client step counts help despite the drift in this example? Proposition 4.1's isotropic setting does not seem to cover this case.
  2. How large is βtstop\beta_t^{\mathrm{stop}} for TCS, and how does it affect convergence?
  3. How does TCS compare with a rule that ignores direction and stops when ∥di,h∥≤(1+κ)∥ri∥/2\|d_{i,h}\|\le(1+\kappa)\|r_i\|/2? Please keep the update budget, step limits, and aggregation the same. A discussion or comparison with FedVeca and FLASH's local stopping rule would also be helpful.
5 comments
AuthorsOpenReview ↗

Response to Reviewer idEn (1/3)

We thank the reviewer for the concrete counterexample and for asking us to clarify stopping bias and the contribution of directional information. We have addressed these concerns through a reproduced counterexample, additional theoretical results, controlled ablations, and expanded baseline comparisons.

1. Unequal local counts and deterministic drift (Q1)
We sincerely thank Reviewer idEn for providing this insightful counterexample. We have reproduced the counterexample in Appendix A.1.4 (lines 782–802) and incorporated its implications into Section 4.1 (lines 293–299). With exact gradients, TCS selects H=(78,44,56)H=(78,44,56) and yields F(θ+)−F(0)=2.82679×10−5>0F(\theta^+)-F(0)=2.82679\times10^{-5}>0. Although the initial client gradients cancel at the global optimum, unequal local counts apply different curvature filters, so the returned updates need not cancel. This is deterministic local drift, distinct from stopping-induced stochastic bias.

Note that Proposition 4.1 concerns an isotropic asymptotic regime. To identify a finite-step setting in which unequal counts preserve descent, Proposition A.3 assumes exact gradients, full participation, a shared minimizer, pairwise commuting positive-definite Hessians, and 0<ηLHmax⁡≤10<\eta L H_{\max}\le1. Under these conditions, global descent holds for every realized count pattern, including unequal counts (lines 805–809). The example in lines 823–826 returns H=(30,25,28)H=(30,25,28) and decreases the objective from 6.166667 to 3.200234.

For more general objectives, Proposition 4.2 separates local alignment, client disagreement, and reference error to characterize when the aggregate update gives global descent (Section 4.2, lines 308–323).

2. Stopping-induced bias and convergence (Q2)
We agree that unbiased stochastic gradients need not produce unbiased adaptively stopped updates. Hence, we add Lemma A.7 to bound the stopping-induced bias while retaining the dependence between adaptive counts and accumulated gradient noise:

\leq C_s\sigma_t,\qquad C_s=\frac{\sqrt{H_{\max}}}{2H_{\min}}(q-q^{-1}), \quad q=\frac{H_{\max}}{H_{\min}}.$$ In Appendix A.4.3, we also provide a tighter expression involving the conditional variability of $\bar H_t/H_i$ (Eq. (25), lines 920–939). For $H_{\min}=10$ and $H_{\max}=100$, the conservative coefficient is $C_s=4.95$. Further, we prepare Corollary A.8, which identifies sufficient conditions for learning rate-weighted expected stationarity, where bounded conditional gradient second moments, fixed finite count bounds, $\sum_t\eta_t=\infty$, $\sum_t\eta_t^2<\infty$, $\sum_t\eta_t\sigma_t^2<\infty$, and summable learning rate-weighted squared sampling bias under partial participation. Note that these imply $$\frac{\sum_{t<T}\eta_t\,\mathbb E\|\nabla F(\theta_t)\|^2} {\sum_{t<T}\eta_t}\longrightarrow 0.$$ We place the statement in Appendix A.4.4 (lines 943–955), with an admissible learning rate/noise schedule in lines 972–977. Also, we summarize in Section 4.3 on how these additional conditions permit stationarity while keeping local counts adaptive and unequal (lines 356–362). We deeply appreciate the reviewer's insightful comments that cleanly sharpen and enhance our analyses.
AuthorsOpenReview ↗

Response to Reviewer idEn (2/3)

3. Directional ablation and related methods (Q3)
We added a Norm-Only control that stops when ∥di,h∥≤(1+κ)∥ri∥/2\|d_{i,h}\|\le(1+\kappa)\|r_i\|/2, using the same reference, local-step bounds, update budget, optimizer, aggregation, and paired seeds as TCS. The control is defined in Section 5.3 (lines 463–465) and Appendix B.3 (lines 1059–1060). Tables 4 and 5 report the five-seed FEMNIST/StackOverflow comparisons and the three-seed MNIST comparisons, respectively.

At κ=0.5\kappa=0.5, the comparisons are:

  • FEMNIST: TCS reaches 82.42 ± 1.54% versus Norm-Only's 78.53 ± 2.80%, using 1,849 versus 1,438 rounds.
  • MNIST, α=0.01\alpha=0.01: TCS gains 1.50 percentage points with 12.4 additional rounds.
  • MNIST, α=0.1\alpha=0.1: TCS has a difference of −0.08 points with 76.3 additional rounds.
  • MNIST, α=1\alpha=1: TCS gains 0.20 points with 191.4 additional rounds.

These results support checking direction as an additional local signal, with an accuracy benefit that varies across partitions. A magnitude-only rule misses directional changes that retain substantial update magnitude.

However, both rules compare updates with each client's own reference. For example, clients could remain locally consistent while moving in different directions, so local consistency alone does not establish agreement across clients. In this context, Figure 2 identifies which signal changes a stopping decision, while Tables 4–5 measure the endpoint effect of enabling direction over training. The largest MNIST accuracy gain occurs where direction-dependent stops are least frequent.

We further discuss this distinction in Section 5.3 (lines 469–485), Appendix B.4 (lines 1073–1079), and Appendix B.5 (lines 1207–1220).

MNIST
(each cell: average accuracy ± standard deviation / average NLL / average communication rounds)

Methodα=0.01α=0.1α=1
Norm-Only κ=0.182.30 ± 2.21 / 0.550 / 372.094.89 ± 0.65 / 0.158 / 406.097.02 ± 0.22 / 0.093 / 363.0
Norm-Only κ=0.389.34 ± 2.65 / 0.338 / 443.395.82 ± 0.42 / 0.130 / 500.797.17 ± 0.18 / 0.089 / 455.7
Norm-Only κ=0.590.29 ± 1.31 / 0.295 / 525.395.92 ± 0.19 / 0.125 / 604.797.24 ± 0.29 / 0.087 / 556.3
Norm-Only κ=0.789.62 ± 5.41 / 0.336 / 626.796.36 ± 0.56 / 0.117 / 721.397.41 ± 0.07 / 0.080 / 682.3
Norm-Only κ=0.992.91 ± 0.42 / 0.224 / 779.096.33 ± 0.09 / 0.116 / 853.397.51 ± 0.08 / 0.081 / 828.0
TCS κ=0.188.17 ± 2.15 / 0.373 / 377.795.35 ± 0.37 / 0.145 / 453.397.30 ± 0.21 / 0.085 / 512.3
TCS κ=0.389.68 ± 2.53 / 0.326 / 451.795.98 ± 0.35 / 0.129 / 566.797.47 ± 0.16 / 0.082 / 642.3
TCS κ=0.591.79 ± 1.22 / 0.252 / 537.795.84 ± 0.66 / 0.131 / 681.097.44 ± 0.13 / 0.082 / 747.7
TCS κ=0.791.76 ± 2.13 / 0.252 / 649.096.46 ± 0.15 / 0.113 / 799.097.54 ± 0.09 / 0.080 / 841.3
TCS κ=0.993.63 ± 0.41 / 0.201 / 794.796.39 ± 0.22 / 0.113 / 899.797.49 ± 0.15 / 0.077 / 908.0

FEMNIST and StackOverflow
(each cell: average accuracy (standard deviation) / average communication rounds)

κFEMNIST Norm-OnlyFEMNIST TCSStackOverflow Norm-OnlyStackOverflow TCS
0.174.54 (1.47) / 83578.13 (2.87) / 1,088.817.61 (0.31) / 150.417.35 (0.35) / 296
0.379.76 (1.97) / 1,116.481.00 (2.21) / 1,463.417.53 (0.65) / 24618.25 (0.23) / 448
0.578.53 (2.80) / 1,43882.42 (1.54) / 1,84917.89 (0.41) / 379.217.94 (0.44) / 634.2
0.780.03 (2.14) / 1,78582.04 (0.79) / 2,154.418.02 (0.32) / 561.218.51 (0.32) / 829
0.982.77 (1.22) / 2,147.882.72 (1.81) / 2,27218.21 (0.41) / 80618.28 (0.42) / 909
AuthorsOpenReview ↗

Response to Reviewer idEn (3/3)

We also expanded Table 1 and Section 2 to clarify the differences from FedVeca and FLASH (lines 102–109).

First, FedVeca selects client-specific step counts using prior-round loss and gradient statistics, whereas TCS decides during local training using existing SGD updates. FedVeca's published protocol exchanges information across all clients, thus we discuss its information requirements under partial participation without introducing a numerical adaptation or comparison.

Second for FLASH, we compare its client-side validation-loss stopping component under common SGD and server aggregation. Its stopping split is disjoint from local training data, and the additional validation forwards are outside the SGD-update budget (Appendix B.3, lines 1055–1058).

In the five-seed main comparison at the common κ=0.7\kappa=0.7, TCS achieves:

  • StackOverflow: 18.51% accuracy in 829 rounds versus FLASH's 18.38% in 943 rounds.
  • CIFAR-100/50: 41.61% accuracy in 2,599 rounds versus FLASH's 39.31% in 2,768 rounds. These results appear in Table 2 and are discussed in Section 5.2 (lines 420–426 and 429–431).

They support TCS's practical contribution: client-specific stopping within each round using existing SGD updates, without additional model evaluations or communication during local training.

For convenience, we reproduce the complete fixed-work comparison from manuscript Table 2 below.

Main comparison under total fixed work
(each cell: average accuracy (standard deviation) / average communication rounds)

MethodFEMNISTStackOverflowCIFAR-100/20CIFAR-100/50CIFAR-100/100
Fixed-1083.26 (0.29) / 2,50018.40 (0.20) / 1,00038.02 (1.22) / 3,00041.68 (0.74) / 3,00043.72 (0.55) / 3,000
Fixed-2080.09 (2.77) / 1,25018.31 (0.31) / 50033.68 (0.37) / 1,50038.66 (0.79) / 1,50042.17 (0.83) / 1,500
Fixed-5076.16 (1.15) / 50018.17 (0.37) / 20026.14 (0.93) / 60030.25 (1.02) / 60035.02 (0.98) / 600
Fixed-10068.16 (2.75) / 25017.91 (0.13) / 10020.59 (1.02) / 30025.19 (0.46) / 30029.72 (0.42) / 300
Mills70.30 (3.59) / 336.417.90 (0.48) / 12319.62 (1.26) / 32924.90 (0.70) / 34528.89 (0.36) / 358
AdaComm70.12 (2.71) / 316.617.86 (0.16) / 11719.98 (0.24) / 31224.70 (0.26) / 32028.98 (0.41) / 342
QSR80.24 (1.94) / 2,31218.04 (0.27) / 92537.14 (0.97) / 2,77540.20 (0.85) / 2,77543.45 (0.17) / 2,775
ALT68.21 (2.93) / 250.617.94 (0.34) / 10128.35 (2.44) / 1,10134.02 (0.73) / 86535.57 (0.79) / 638
FLASH82.36 (0.77) / 2,377.818.38 (0.19) / 94330.33 (0.86) / 2,74739.31 (0.66) / 2,76843.19 (0.54) / 2,791
TCS κ=0.178.13 (2.87) / 1,088.817.35 (0.35) / 29633.47 (0.52) / 1,57036.55 (0.94) / 1,54641.21 (0.21) / 1,501
TCS κ=0.381.00 (2.21) / 1,463.418.25 (0.23) / 44837.35 (0.38) / 1,92437.45 (0.78) / 1,89842.36 (0.64) / 1,766
TCS κ=0.582.42 (1.54) / 1,84917.94 (0.44) / 634.236.53 (1.01) / 2,28039.61 (0.62) / 2,26443.53 (0.68) / 2,272
TCS κ=0.782.04 (0.79) / 2,154.418.51 (0.32) / 82937.90 (0.41) / 2,59841.61 (0.82) / 2,59941.75 (0.32) / 2,600
TCS κ=0.982.72 (1.81) / 2,27218.28 (0.42) / 90937.26 (0.36) / 2,72740.06 (0.35) / 2,72741.74 (0.31) / 2,727

Once again, we would like to express our deep appreciation for the reviewer's constructive feedback, which substantially improved our manuscript.

Reviewer_M31rOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Trajectory Consistency-based Stopping (TCS), an adaptive local-step selection method for federated learning. Rather than using a fixed number of local updates for all clients, TCS establishes a reference from the initial local updates and determines whether to continue training by comparing the direction and magnitude of the current average update with this reference. A robustness parameter controls how conservatively local training is stopped. The method also employs work-normalized aggregation to account for differences in the number of local steps across clients. The paper provides theoretical analyses of curvature-dependent stopping, the relationship between local update consistency and global descent, and a nonconvex convergence bound under partial client participation that explicitly accounts for sampling, local drift, and stopping-induced bias and variance. Experiments on FEMNIST, CIFAR-100, and StackOverflow compare TCS against fixed-step and adaptive scheduling baselines under a common local-update budget. With κ = 0.7, TCS achieves accuracy comparable to Fixed-10 while reducing communication rounds by 24.57%, 22.77%, and 36.13%, respectively. Additional MNIST experiments investigate how the selected local-step counts and stopping behavior change with data heterogeneity. Overall, the results suggest that local trajectory consistency is a promising signal for allocating computation between communication rounds.

Critical strengths

Strength 1: A simple, client-specific stopping rule with low additional overhead. TCS uses information already generated by local SGD to determine when each client should stop updating. Unlike approaches that require additional local-global representation comparisons, the proposed criterion does not require extra model evaluations or communication during local training. Its ability to adapt local computation to individual client trajectories is a practical advantage in heterogeneous federated learning.

Strength 2: A combination of theoretical analysis and budget-controlled empirical evaluation. The paper investigates both the optimization implications of trajectory-based stopping and its empirical communication–accuracy trade-off. Experiments across three main benchmarks, together with MNIST heterogeneity and stopping-behavior analyses, provide evidence beyond a single dataset or setting. Under the reported local-update budget, TCS substantially reduces communication rounds while maintaining accuracy close to the Fixed-10 baseline.

Critical weaknesses

Weakness 1: Limited evidence for empirical robustness and hyperparameter sensitivity. The main benchmark results are averaged over only three paired seeds, which provides limited evidence about the stability of the reported improvements. Moreover, the primary experiments use κ = 0.7, while the detailed sensitivity analysis is conducted on MNIST rather than across the three main benchmarks. The paper does not sufficiently clarify how this setting was selected or establish that the observed trade-off is robust to reasonable changes in κ and the base-step count. More extensive sensitivity analyses and uncertainty estimates would strengthen the empirical claims.

Weakness 2: The convergence guarantee remains conditional on unresolved bias and variance terms. Theorem 4.3 provides a nonconvex stationarity bound, but the paper explicitly notes that Assumptions 4.1–4.4 alone do not guarantee that the weighted bias and variance contributions vanish. In particular, adaptive stopping introduces dependence between the selected local-step counts and the observed stochastic updates. Consequently, the theorem does not by itself establish convergence under the stated assumptions without additional conditions on these terms. Clarifying sufficient conditions under which the stopping-induced bias can be controlled would strengthen the theoretical contribution.

Questions for authors

Question 1 — Hyperparameter selection and empirical robustness How were κ = 0.7 and the base-step count selected for the three main benchmarks? Were these settings fixed in advance or chosen using validation results? Please clarify the selection protocol and, if feasible, provide sensitivity analyses on FEMNIST, CIFAR-100, and StackOverflow to demonstrate that the communication–accuracy trade-off is not specific to one hyperparameter configuration.

Question 2 — Practical implications of the convergence bound Theorem 4.3 retains sampling, local-drift, and stopping-induced bias terms, while the paper notes that the stated assumptions alone do not guarantee that the relevant bias and variance contributions vanish. Can the authors identify additional sufficient conditions under which the stopping-induced bias is controlled and the bound implies stationarity? This clarification would help establish the scope of the theoretical guarantee.

Question 3 — Statistical reliability of the main results The main benchmark comparison reports means and standard deviations over three paired seeds. Can the authors provide additional-seed results or paired uncertainty estimates for the accuracy differences relative to Fixed-10? This would help determine whether the observed improvements on CIFAR-100 and the near-equivalent performance on FEMNIST and StackOverflow are reproducible.

4 comments
AuthorsOpenReview ↗

Response to Reviewer M31r (1/2)

We thank the reviewer for recognizing TCS's client-specific, low-overhead design and for identifying areas that require stronger evidence. We have addressed all three concerns through broader hyperparameter sensitivity analyses, explicit sufficient conditions for stationarity, and five-seed empirical comparisons.

1. Hyperparameter selection and robustness (W1, Q1)

The main comparison uses Hmin⁡=10H_{\min}=10 and Hmax⁡=100H_{\max}=100, with the former aligned with our epoch-scale batch calibration. The original experiments used a common κ=0.7\kappa=0.7 across benchmarks rather than separate dataset-specific configurations.

To assess robustness beyond this setting, we conducted five-seed sweeps over κ∈{0.1,0.3,0.5,0.7,0.9}\kappa\in\{0.1,0.3,0.5,0.7,0.9\} and Hmin⁡∈{10,20,50}H_{\min}\in\{10,20,50\}, with Hmax⁡=100H_{\max}=100, on FEMNIST, StackOverflow, and CIFAR-100 under three label-heterogeneity conditions (20/50/100 allocated classes per client). All CIFAR-100 conditions retain the same 100-class prediction task.

We add Figure 3 to report the complete accuracy grids (15 cells per dataset), including less favorable configurations. These results establish the range of achievable accuracy–communication trade-offs and clarify sensitivity to both parameters. In particular, Hmin⁡H_{\min} affects not only the minimum local computation but also the reference used by TCS. We therefore do not claim that a single configuration is universally optimal.

2. Sufficient conditions for convergence (W2, Q2)

We agree that Assumptions 4.1–4.4 alone do not guarantee stationarity. The revision makes this limitation explicit and adds two theoretical results:

  • Lemma A.6 bounds the stopping-induced bias as ∥βtstop∥≤Csσt\|\boldsymbol{\beta}_t^{\mathrm{stop}}\|\leq C_s\sigma_t, explicitly accounting for the dependence between adaptive stopping counts and stochastic updates.
  • Corollary A.7 establishes learning-rate-weighted expected stationarity under additional conditions, including bounded local-step counts and conditional gradient second moments, suitable learning-rate/noise summability, and summable learning-rate-weighted squared sampling bias.

These are additional sufficient conditions, not consequences of the original assumptions. The revised Section 4.3 and Appendix A.4.3–A.4.4 therefore clarify precisely when the convergence bound implies stationarity.

3. Additional seed results and communication efficiency with comparable performance (Q3)

We expanded the main comparisons from three to five paired seeds, reporting mean accuracy and sample standard deviation under matched local-update budgets within each benchmark.

The following results retain the original common setting κ=0.7\kappa=0.7 rather than selecting a different threshold for each task.

BenchmarkFixed-10 accuracy (%)TCS accuracy (%)Δ\Delta accuracy (pp)Fewer rounds
FEMNIST83.26 ± 0.2982.04 ± 0.79−1.2213.8%
StackOverflow18.40 ± 0.2018.51 ± 0.32+0.1117.1%
CIFAR-100/2038.02 ± 1.2237.90 ± 0.41−0.1213.4%
CIFAR-100/5041.68 ± 0.7442.61 ± 0.82+0.9313.4%
CIFAR-100/10039.90 ± 0.8139.85 ± 0.29−0.0513.3%

The broader sweep also identifies favorable operating points. At κ=0.5\kappa=0.5, TCS achieves:

  • FEMNIST: 82.42 ± 1.54% accuracy with 26.0% fewer rounds than Fixed-10, an accuracy difference of only −0.84 percentage points.
  • CIFAR-100/100: 41.53 ± 0.72% accuracy with 24.3% fewer rounds, exceeding Fixed-10's mean accuracy by 1.63 percentage points.

Overall, we thank the reviewer again as the additional experiments support TCS's intended benefit more: adapting each client's local computation online without prior tuning efforts, with saved communication while preserving competitive accuracy. The theoretical additions also clarify the sufficient conditions for stationarity without overstating their applicability to the empirical settings.

AuthorsOpenReview ↗

Response to Reviewer M31r (2/2)

For your reference, we also provide the five paired seeds full comparison table below. Each cell indicates {mean accuracy (standard deviation) / average communication rounds}. We boldface the best fixed and adaptive baselines each per dataset, as well as the best TCS. (per cell, {higher is better; lower is better})

(each cell: average accuracy (standard deviation) / average communication rounds)

MethodFEMNISTStackOverflowCIFAR-100/20CIFAR-100/50CIFAR-100/100
Fixed-1083.26 (0.29) / 2,50018.40 (0.20) / 1,00038.02 (1.22) / 3,00041.68 (0.74) / 3,00043.72 (0.55) / 3,000
Fixed-2080.09 (2.77) / 1,25018.31 (0.31) / 50033.68 (0.37) / 1,50038.66 (0.79) / 1,50042.17 (0.83) / 1,500
Fixed-5076.16 (1.15) / 50018.17 (0.37) / 20026.14 (0.93) / 60030.25 (1.02) / 60035.02 (0.98) / 600
Fixed-10068.16 (2.75) / 25017.91 (0.13) / 10020.59 (1.02) / 30025.19 (0.46) / 30029.72 (0.42) / 300
Mills70.30 (3.59) / 336.417.90 (0.48) / 12319.62 (1.26) / 32924.90 (0.70) / 34528.89 (0.36) / 358
AdaComm70.12 (2.71) / 316.617.86 (0.16) / 11719.98 (0.24) / 31224.70 (0.26) / 32028.98 (0.41) / 342
QSR80.24 (1.94) / 2,31218.04 / (0.27) / 92537.14 (0.97) / 2,77540.20 (0.85) / 2,77543.45 (0.17) / 2,775
ALT68.21 (2.93) / 250.617.94 (0.34) / 10128.35 (2.44) / 1,10134.02 (0.73) / 86535.57 (0.79) / 638
FLASH82.36 (0.77) / 2,377.818.38 (0.19) / 94330.33 (0.86) / 2,74739.31 (0.66) / 2,76843.19 (0.54) / 2,791
TCS κ=0.178.13 (2.87) / 1,088.817.35 (0.35) / 29633.47 (0.52) / 1,57036.55 (0.94) / 1,54641.21 (0.21) / 1,501
TCS κ=0.381.00 (2.21) / 1,463.418.25 (0.23) / 44837.35 (0.38) / 1,92437.45 (0.78) / 1,89842.36 (0.64) / 1,766
TCS κ=0.582.42 (1.54) / 1,84917.94 (0.44) / 634.236.53 (1.01) / 2,28039.61 (0.62) / 2,26443.53 (0.68) / 2,272
TCS κ=0.782.04 (0.79) / 2,154.418.51 (0.32) / 82937.90 (0.41) / 2,59841.61 (0.82) 2,59941.75 (0.32) / 2,600
TCS κ=0.982.72 (1.81) / 2,27218.28 (0.42) / 90937.26 (0.36) / 2,72740.06 (0.35) / 2,72741.74 (0.31) / 2,727

We would like to thank the reviewer again for positive evaluation, which enhanced our manuscript.

Thank you for the clarifications. The authors have adequately addressed my questions, and I appreciate their efforts. I will keep my original rating.

Thank you for your thoughtful feedback and careful consideration of our response. We appreciate your time and constructive comments, which have certainly helped us improve the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.