acceptodds
Under review as a conference paper at ICLR 2027

TRUSS: Transmission-Reduced Unified-State Scaffolding under Non-Stationary Client Participation

Abstract

Heterogeneous client objectives and unequal, time-varying availability create coupled sources of error in federated optimization: local drift and participation bias. Correcting both while preserving the speedup from local computation and keeping communication and memory costs low is a central challenge. We introduce Transmission-Reduced Unified-State Scaffolding (TRUSS), which addresses both with a single persistent, model-sized memory per client. The difference between this memory and the current server model supplies the correction applied at every local step. Each active client therefore exchanges one model-sized vector in each direction per round, matching FedAvg's per-round communication, and the server stores only the global model. For smooth non-convex objectives under independent participation whose probabilities may vary across clients and rounds, we establish an stationarity bound, yielding linear speedup in the number of clients and local steps , with no bounded-gradient-heterogeneity assumption and no knowledge of client inclusion probabilities or importance weighting. Experiments demonstrate the effectiveness and communication efficiency of TRUSS.

Primary area:
optimization
Keywords:
Federated Learning, Variance Reduction, Arbitrary Client Participation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tJDLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The manuscript investigates the federated optimization problem under the conditions of heterogeneous client objective functions and uneven client availability, as well as non-stationary client participation. It proposes a unified state-based approach for communication reduction, named TRUSS, which simultaneously addresses these two issues by maintaining only one persistent state memory of equal size to the model on each client. In terms of experiments, the authors tested five client availability patterns on CIFAR-10, CIFAR-100, and SVHN to verify the communication efficiency and convergence robustness of TRUSS.

Critical strengths

The proposed method achieves high performance without significantly increasing overhead. Furthermore, the manuscript is comprehensive in terms of workload, with rich experimental and theoretical explanations.

Critical weaknesses

The research question is relatively conventional, and there have already been many excellent works proposed in this area. Furthermore, although the manuscript contains numerous experiments, it lacks some necessary discussions, such as those related to federated learning.

Questions for authors

(1) Regarding the experimental aspect of federated learning, the experiments in the full paper are only fixed at N=10, K=50. Different training steps and client numbers should be set, especially for large-scale client verification.

(2) When the client comes back online after experiencing a long period of complete offline, will the value of ‖mir−1−xr−1‖/h‖m_i^{r-1}-x^{r-1} ‖/h be too large, potentially causing the local model to diverge?

(3) Why do FedAWE and SGFocus experience catastrophic failures on CIFAR-100 and SVHN in Table 2, even under near-IID conditions? Furthermore, it is suggested to supplement Table 1 with experimental comparisons of the closest competing method, FedSUM-CR, under the same non-stationary scenario?

(4) In Figure 1, SCAFFOLD and SCALLION have almost the same accuracy as TRUSS during training. Why are SCAFFOLD and SCALLION only achieving 59.2% and 67.1% respectively under the same settings in Table 2?

6 comments
AuthorsOpenReview ↗

Response to Reviewer tJDL (1/3)

Thank you for reading our paper and providing constructive feedback. Our responses are below.

Practical significance and contribution

Thank you for recognizing the efficiency of TRUSS and the breadth of its theoretical and experimental evaluation.

Novelty of the research question.

We agree that client drift and partial participation have each been studied extensively, and Section 2 and Table 1 compare TRUSS with many of these works. However, data heterogeneity, irregular client participation, and communication and memory overhead remain major barriers between federated learning methods and their practical, large-scale deployment. A recent account of production federated learning systems reports that coordinating training across devices that differ in compute, bandwidth, availability, and data hinders broader adoption, and that modern large models are orders of magnitude beyond what classical cross-device federated learning has handled (Daly et al., 2024). Each barrier has a direct optimization cost. Heterogeneous data cause local drift, which limits the benefit of local computation. Irregular, time-varying availability biases the model toward frequently available clients and can change the objective that is actually optimized. Communication overhead is widely identified as the main bottleneck for real-world deployment, because large model updates must be exchanged repeatedly (Le et al., 2024). The same issues now arise beyond classical federated learning: federated-averaging variants with many local steps are used to train large language models across poorly connected, geographically distributed clusters whose available resources change during training (Douillard et al., 2023). Our experiments are run on such a geographically distributed deployment, with nodes in different U.S. states (Section 7).

These barriers also interact. Methods that correct drift and participation bias typically pay for these corrections with exactly the overhead that limits deployment: extra control vectors in communication, additional client state, or auxiliary server memory. For large models, parameter transmission can dominate the per-round cost. Per-client memory is limited on resource-constrained devices. Server storage that grows with the number of clients, as in MIFA, becomes prohibitive at scale. Addressing all three barriers at once, rather than trading one for another, is therefore essential for deploying federated learning in practice.

Concretely, what remains open is achieving all of the following at once:

  1. per-step drift correction without a bounded gradient-heterogeneity assumption;
  2. convergence under unequal, time-varying participation, with no bound on inactivity length and no knowledge of participation probabilities;
  3. linear speedup in both the number of clients and the number of local steps;
  4. FedAvg-level cost: one model-sized vector in each direction per round, only the global model stored at the server, and one model-sized vector per client.

Each existing method gives up at least one of these properties (Table 1):

  • SCAFFOLD and SCALLION satisfy (1) and (3), but assume uniform sampling and require extra communication and server state.
  • MIFA, FedAU, and FedAWE target irregular availability, but rely on heterogeneity-dependent assumptions and do not correct local steps. MIFA also stores NN additional model-sized vectors at the server.
  • SG-FOCUS removes the heterogeneity bound, but assumes stationary participation and does not establish local-step speedup.
  • Amplified SCAFFOLD requires a participation-balance condition over windows of rounds.
  • FedSUM-CR attains two-vector communication, but requires bounded inactivity, stores a second vector at every client and an auxiliary vector at the server, and its rates depend on the worst-case participation delay.

To our knowledge, TRUSS is the first method to satisfy all four properties. The enabling idea is that a single memory per client performs both corrections: the gap between a client's memory and the current server model supplies the drift correction, and it also records the server's movement while the client was absent (Section 4). On the analysis side, analysis-only backward participation weights establish descent at the actual server model without requiring unbiased server updates. This is what allows the guarantee to cover unequal, time-varying participation probabilities (Sections 5.2–5.3).

References.

  • Daly et al. (2024). Federated Learning in Practice: Reflections and Projections. IEEE TPS-ISA. arXiv:2410.08892

  • Le et al. (2024). Exploring the Practicality of Federated Learning: A Survey Towards the Communication Perspective. arXiv:2405.20431

  • Douillard et al. (2023). DiLoCo: Distributed Low-Communication Training of Language Models. arXiv:2311.08105

AuthorsOpenReview ↗

Response to Reviewer tJDL (2/3)

Comment 1: Client-count and local-step scaling

Appendix C.2 (Figure 5) extends the original setting to SVHN/CNN experiments with 10–100 clients and KK ranging from 10 to 50 under exponential client participation. Increasing the client count at a fixed participation fraction generally reduces the rounds needed to reach 8080\\% test accuracy. More local steps also reduce rounds, with diminishing gains as KK grows.

Appendix C.2 (Figure 6) distinguishes this from holding the expected active-client count fixed. Its fixed-count slices are derived from an interpolated surface fitted to measured points and suggest increasing round counts as the total client count grows. The evidence establishes these trends on SVHN/CNN up to 100 clients; larger-scale behavior remains outside the tested range.

Comment 2: Stability after prolonged client inactivity

For the SGD updates analyzed in Algorithm 1, a returning client restarts at the current server model. A long absence can enlarge the memory–model discrepancy; the analysis does not assume it is uniformly small. Since h=Kηgηℓh=K\eta_g\eta_\ell, the correction contributes

−ηℓdir−1=xr−1−mir−1Kηg-\eta_\ell d_i^{r-1} =\frac{x^{r-1}-m_i^{r-1}}{K\eta_g}

to each local step. Summing the steps yields mir=xr−1−hg‾irm_i^r=x^{r-1}-h\overline g_i^r, where g‾ir\overline g_i^r averages the current trajectory's raw stochastic gradients. This refresh replaces the old memory, while its influence on the trajectory remains accounted for through g‾ir\overline g_i^r. Intuitively, a larger refresh upon return can compensate for changes accumulated during inactivity and preserve the influence of rarely participating clients.

Appendix A.5 jointly bounds local drift and memory-tracking error. Under Assumptions 1–3 and the step-size conditions of Theorem 1, these bounds establish convergence in expectation for independent participation with pir≥δ>0p_i^r\ge\delta>0, without imposing a maximum inactivity duration. The guarantee therefore accommodates long realized absences, but does not assert that every returning client's update is small.

AuthorsOpenReview ↗

Response to Reviewer tJDL (3/3)

Comment 3: Baseline performance and FedSUM-CR comparison

Table 2 uses strongly non-i.i.d. data partitions with Dirichlet α=0.01\alpha=0.01. “Uniform” refers to client participation probabilities, not the data distribution. Learning rates are selected separately for each method and setting under the same reported communication budget (Appendix B.3).

We hypothesize that SGFocus's poor performance at K=50K=50 partly reflects limited benefit from additional local computation under this budget. Its stochastic non-convex theorem does not establish linear speedup in KK and includes a step-size restriction that tightens with KK (Theorem 6).

FedAWE multiplies a client's update by the number of rounds since its last participation, while its local SGD lacks an explicit drift-correction control variate (Section 5). This can amplify local drift under severe heterogeneity. Consistent with sensitivity to heterogeneity, Appendix C.1 shows CIFAR-10 mean test accuracy increasing from 28.828.8\\% to 78.978.9\\% as α\alpha increases from 0.010.01 to 1010.

We have added FedSUM-CR to Table 2 across all three datasets and five participation patterns under the reported fixed communication budget. Each method is tuned separately using the same grid in Appendix B.3, with selection based on average test accuracy over the final 25 rounds. Under this protocol, TRUSS achieves higher mean test accuracy in all 15 settings. On SVHN/CNN under sine participation, TRUSS achieves 83.0±1.583.0\pm1.5\\%, compared with 35.26±2.4235.26\pm2.42\\% for FedSUM-CR (mean ±\pm standard deviation over three seeds).

We believe FedSUM-CR's retained gradients and temporally averaged corrections may slow adaptation under heterogeneous data and intermittent participation, while TRUSS's correction using accumulated model movement may better support progress within the evaluated budget.

As a sanity check, we also evaluated FedSUM-CR on SVHN/CNN at Dirichlet α=0.1\alpha=0.1, as used in the FedSUM paper. Accuracy improves across all five participation patterns. Agreement with the published curves is a qualitative check because the experimental code for reproducibility is not available.

SVHN/CNN: test accuracy (%, mean ±\pm SD)

Client participationTRUSS (α=0.01\alpha=0.01)FedSUM-CR (α=0.01\alpha=0.01)FedSUM-CR (α=0.1\alpha=0.1)
Uniform84.3±0.884.3\pm0.862.97±7.1262.97\pm7.1280.27±0.0180.27\pm0.01
Stationary83.7±0.083.7\pm0.037.56±3.0737.56\pm3.0768.56±10.3568.56\pm10.35
Staircase84.5±0.184.5\pm0.136.54±2.0436.54\pm2.0449.96±7.0949.96\pm7.09
Sine83.0±1.583.0\pm1.535.26±2.4235.26\pm2.4259.21±1.7359.21\pm1.73
Exponential85.3±0.085.3\pm0.031.40±5.9731.40\pm5.9761.65±7.9661.65\pm7.96

TRUSS requires one persistent algorithmic vector per client and one at the server, compared with two at each for FedSUM-CR (Table 1). The comparison supports higher reported accuracy within the evaluated tuning grid with lower persistent algorithmic storage.

Comment 4: Figure 1 versus Table 2

Table 2 reports accuracy under an aggregate model-payload communication budget of 1000B1000B, where BB is model size, whereas Figure 1 (left) compares accuracy over communication rounds. For method jj, the budget counts all active clients' uplink and downlink model payloads:

Cj(R)=cjB∑r=1R∣Sr∣,cTRUSS=2,cSCAFFOLD=4,cSCALLION=3.C_j(R)=c_jB\sum_{r=1}^R|\mathcal S^r|,\qquad c_{\mathrm{TRUSS}}=2,\quad c_{\mathrm{SCAFFOLD}}=4,\quad c_{\mathrm{SCALLION}}=3.

SCAFFOLD and SCALLION therefore consume the budget more quickly for a given participation sequence.

The 59.259.2\\% and 67.167.1\\% values are averages over the final 25 rounds of the budget-limited runs, with the tuned learning rates listed in Table 3. Similar accuracy at a given round can therefore coexist with different accuracy at a given communication budget. This explains the distinction between the reported metrics.

Reviewer_2cqDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper tries to use a local state at each client in federated learning then use it during the training to offset the client drift and participation bias. The authors present a suit of convergence analysis for their method and evaluate it through benchmark comparisons.

Critical strengths

The idea of the paper is quite interesting and careful design of the local state of the client is shown to work across variety of simulation settings.

The method seems to lead to the most performance gain in the most non-iid data setting (Fig. 2(a)), which makes sense.

The convergence analysis justify the design selections.

The experiments seem to be comprehensive.

Critical weaknesses

There are some issues with the paper that needs to be addressed:

  1. The main text of the paper is extremely hard to follow. Even for someone with knowledge on the theory behind FL, the paper reads extremely heavy and is very difficult to follow. One of the main reasons behind this is that the paper is not telling a cohesive story, instead it is explain the methodology with abrupt jumps across sentences and paragraphs. This undermines the value and contribution of the paper.

  2. It is not clear what is the main advantage of the method compared to gradient tracking techniques. For example, why the gradient tracking techniques are not capable of achieving the properties that the paper claims to achieve for FL. Also, it is not clear what is the convergence acceleration compared to gradient tracking techniques.

Questions for authors
  1. The writing of the paper seems to need a major revamp. I am not sure whether asking for such major change is justified during a short review process as it will majorly alter the presentation of the paper. Nevertheless, can the author explain what their strategy would be for refining the text?

  2. Can the authors explain the advantage over the gradient tracking techniques (both in theory and experiments)?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 2cqD

Thank you for reading our paper and providing constructive feedback. Our responses are below.

Comment 1: Presentation and coherence

Thank you for the candid feedback. We agree that the main text is dense and moves from mechanism to analysis without enough explanation in between. Below we give the storyline in plain terms, with pointers to where each piece appears in the paper.

The problem.

Federated learning faces two errors at once. With heterogeneous data, local steps pull each client toward its own optimum (client drift). With unequal, time-varying availability, frequently available clients dominate the updates (participation bias). SCAFFOLD-type methods correct drift, but they need an extra control vector in communication and server storage, and they assume uniform sampling. Availability-aware methods correct participation bias, but they do not correct the individual local steps. (Section 1, Table 1)

The idea: one memory per client.

Think of each client's memory mim_i as its “vote” in the server model. The server model is always exactly the average of all clients' votes. When a client participates, it replaces its old vote with a fresh one: the current server model minus a step along its new gradient. When a client is absent, its old vote is kept and still counts in the average, so absent clients are never dropped. (Algorithm 1, Eq. (4.2))

Why one vector does two jobs.

When a client returns, it compares its stored vote with the current server model. The gap contains (a) how far the server, i.e., the population, has moved since the client's last visit, and (b) the client's own previous gradient. Together they form a SCAFFOLD-style correction, “global direction minus my own direction,” which removes drift at every local step. For a client returning after a long absence, (a) also carries what the server did while it was away, which compensates for the missed rounds. The gap is computed from the downloaded model, so each round the client downloads one model and uploads one update, as in FedAvg, and the server stores only the model. (Section 4, “Refreshing the memory”)

What the proof has to handle.

Rarely available clients hold older votes, so their average is not an unbiased estimate of a fresh gradient step. The analysis handles this in three steps:

  • (i) Measuring each vote's error relative to where it should be turns data heterogeneity into a tracking error, so no heterogeneity bound is needed (Section 5.1).
  • (ii) An analysis-only reweighting of the votes cancels their unequal ages, so the reweighted trajectory moves along the true global gradient (Section 5.2).
  • (iii) A potential function carries this progress over to the actual server model and decreases every round (Section 5.3).

The result is Theorem 1 and Corollary 1 (Section 6): the rate improves linearly with the number of clients and local steps, and under uniform participation it matches SCAFFOLD's leading term with half of SCAFFOLD's communication.

Revision strategy.

For the final version, we will open Section 4 with the explanation above and revise Section 5 to include more explanation about the intuition behind each step of the analysis with less mathematical detail. Appendix A will provide the full derivations.

Comment 2: Comparison with gradient tracking

Gradient-tracking methods use auxiliary gradient estimates to align local updates with the global objective under data heterogeneity, typically incurring additional communication and memory costs. SCAFFOLD and SCALLION (Table 2) are closely related control-variate baselines. TRUSS unifies local drift correction and compensation for unequal, time-varying participation in one persistent client memory: its difference from the global model corrects local updates, while its retention preserves inactive clients' contributions.

Because the server model equals the average client memory, TRUSS requires no separate server control variate. As shown in Table 1, it exchanges two model-sized vectors per active client per round, compared with four for original SCAFFOLD and three for SCALLION. All three methods store one persistent vector per client; server storage, including the global model, is one vector for TRUSS versus two for both baselines.

The theoretical distinction is that the standard SCAFFOLD and SCALLION analyses assume uniform client sampling, whereas TRUSS establishes convergence under independent, unequal, time-varying participation, without a bounded gradient-heterogeneity assumption and with client and local-step speedup in the leading term.

Empirically, Figure 1 shows comparable convergence in rounds, with TRUSS reaching the same accuracy at lower communication cost. Under Table 2's fixed communication budget, TRUSS achieves higher mean test accuracy than both baselines in all 15 dataset–participation settings.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer,

We are gently checking in to see if the simplified storyline and our strategy for revising the presentation, alongside the comparisons with gradient tracking, resolved your questions. We are happy to discuss further.

Best,

Authors

I would like to thank the authors for their responses. Unfortunately, the responses are not convincing. First, the comparison with more recent federated gradient tracking methods is still missing (even if they use more communication overhead, these methods can converge faster, so their overall resource overhead could be smaller). Second, I maintain my judgement that the paper is not easy to follow and understand as someone who works in this area. Thus, I maintain my original rating.

We thank the reviewer for the follow-up, and we apologize that our first response did not make this clear: it discussed only SCAFFOLD and SCALLION, but our experiments also include SG-FOCUS (NeurIPS 2025), a recent federated gradient-tracking method, and FedSUM-CR (arXiv 2025). All four baselines appear in Figure 1 and Table 2.

Overall resource cost. The reviewer notes that methods with higher per-round communication may converge faster and therefore use fewer resources overall. Our experiments test this directly, and it is not what we observe. TRUSS reaches comparable test accuracy in a similar number of rounds (Figure 1, left), so the baselines do not converge faster. Because TRUSS reconstructs its drift correction locally, it transmits no separate correction vector and reaches the same accuracy with fewer total bytes (Figure 1, right). Under the fixed communication budget in Table 2, TRUSS achieves higher mean test accuracy than each of the four baselines in all 15 settings.

Theory. As in the analyses of other gradient-tracking methods, our convergence rate requires no bound on gradient heterogeneity. Under uniform participation, it matches SCAFFOLD's leading term with half the communication. Unlike analyses that assume uniform client sampling, it also holds under independent, unequal, time-varying participation.

Presentation. We take this concern seriously. As outlined in our first response, we will open Section 4 with the plain-language storyline and restructure Section 5 around the intuition for each step of the analysis, moving the full derivations to Appendix A. If the reviewer can point to the passages that were hardest to follow, we will prioritize them.

Reviewer_RCgtOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies client drift and unequal, time-varying participation in federated learning using one persistent memory vector per client. It matches FedAvg’s communication and server storage costs, while providing nonconvex convergence guarantees with client and local-step speedup under independent participation. Experiments demonstrate competitive accuracy and communication efficiency across several vision benchmarks.

Critical strengths
  • TRUSS has a simple and resource-efficient design. TRUSS uses a single persistent memory vector per client to jointly correct local drift and compensate for unequal participation. It matches FedAvg’s per-round communication and server storage requirements, making the design practically appealing.
  • The analysis accommodates unequal, time-varying participation probabilities without requiring a bounded gradient-heterogeneity assumption. Under independent participation with a positive probability lower bound, the leading convergence term supports linear speedup in both client count and local steps.
Critical weaknesses
  • FedSUM-CR appears to be a particularly relevant baseline, as Table 1 highlights TRUSS’s communication and memory advantages over it. However, the main experiments do not directly compare the two methods. Such a comparison would help clarify what TRUSS adds beyond existing approaches.
  • The experiments fix N=10N=10 and K=50K=50, leaving it unclear whether increasing either parameter reduces the number of communication rounds needed to reach the same target. It would also help to distinguish between holding the client participation fraction fixed and holding the expected number of active clients fixed, as these represent different experimental settings.
Questions for authors
  • How does the speedup translate into practice? Can you vary NN and KK and report the communication rounds and wall-clock time needed to reach the same target? For NN, please distinguish fixed participation fraction from fixed expected active-client count.
  • How does TRUSS compare directly with FedSUM-CR? Since this is a close baseline in Table 1, could you compare accuracy, communication cost, and memory usage under the same experimental conditions?
  • How should users choose the learning rates without participation information? The updates do not require participation probabilities, but the theoretical step sizes depend on their lower bound δ\delta. How sensitive is practical performance when this bound is unknown or very small?
4 comments
AuthorsOpenReview ↗

Response to Reviewer RCgt (1/2)

Thank you for reading our paper and providing constructive feedback. Our responses are below.

Comment 1: Direct comparison with FedSUM-CR

We have added FedSUM-CR to Table 2 across all three datasets and five participation patterns under the reported fixed communication budget. Each method is tuned separately using the same grid in Appendix B.3, with selection based on average test accuracy over the final 25 rounds. Under this protocol, TRUSS achieves higher mean test accuracy in all 15 settings. On SVHN/CNN under sine participation, TRUSS achieves 83.0±1.583.0\pm1.5\\%, compared with 35.26±2.4235.26\pm2.42\\% for FedSUM-CR (mean ±\pm standard deviation over three seeds).

We believe FedSUM-CR's retained gradients and temporally averaged corrections may slow adaptation under heterogeneous data and intermittent participation, while TRUSS's correction using accumulated model movement may better support progress within the evaluated budget.

As a sanity check, we also evaluated FedSUM-CR on SVHN/CNN at Dirichlet α=0.1\alpha=0.1, as used in the FedSUM paper. Accuracy improves across all five participation patterns. Agreement with the published curves is a qualitative check because the experimental code for reproducibility is not available.

SVHN/CNN: test accuracy (%, mean ±\pm SD)

Client participationTRUSS (α=0.01\alpha=0.01)FedSUM-CR (α=0.01\alpha=0.01)FedSUM-CR (α=0.1\alpha=0.1)
Uniform84.3±0.884.3\pm0.862.97±7.1262.97\pm7.1280.27±0.0180.27\pm0.01
Stationary83.7±0.083.7\pm0.037.56±3.0737.56\pm3.0768.56±10.3568.56\pm10.35
Staircase84.5±0.184.5\pm0.136.54±2.0436.54\pm2.0449.96±7.0949.96\pm7.09
Sine83.0±1.583.0\pm1.535.26±2.4235.26\pm2.4259.21±1.7359.21\pm1.73
Exponential85.3±0.085.3\pm0.031.40±5.9731.40\pm5.9761.65±7.9661.65\pm7.96

TRUSS requires one persistent algorithmic vector per client and one at the server, compared with two at each for FedSUM-CR (Table 1). The comparison supports higher reported accuracy within the evaluated tuning grid with lower persistent algorithmic storage.

Comment 2: Client-count and local-step scaling

Appendix C.2 (Figure 5) reports new SVHN/CNN experiments under exponential client participation, measuring rounds to 8080\\% test accuracy with 10–100 clients and KK ranging from 10 to 50. More clients at a fixed participation fraction and higher participation at a fixed client count generally reduce the required rounds. Increasing KK also reduces rounds, with saturation at larger local-step budgets.

Appendix C.2 (Figure 6) shows fixed-count slices derived from an interpolated surface fitted to measured points. Over the plotted range, these slices suggest that increasing the total client count requires more rounds when the expected active-client count is fixed. Holding the active-client count fixed lowers mean per-client participation as NN grows. This differs from the fixed-participation-parameter regime underlying the leading theoretical speedup term.

These experiments report communication rounds. The translation to elapsed time depends on computation, transmission, and synchronization costs. Fewer rounds can substantially shorten training when communication dominates (as is usually the case for large models).

AuthorsOpenReview ↗

Response to Reviewer RCgt (2/2)

Comment 3: Learning-rate selection without participation information

The dependence on δ\delta in our sufficient convergence conditions is consistent with the literature, where theoretical step sizes also depend on participation probabilities or inactivity, as summarized below.

Method / sourceTheoretical step sizes: participation dependence
TRUSS (Corollary 1)ηg=δN\eta_g=\delta\sqrt{N}, \displaystyle \eta_\ell=\min\\!\left\\{ \frac{1}{\beta\delta\sqrt{\delta KR}}, \frac{\delta}{200\beta\Lambda K\sqrt{N}}\right\\}.
FedSUM / FedSUM-CR (Theorem 4.1)ηg=τmax⁡−1/2\eta_g=\tau_{\max}^{-1/2}, \displaystyle \eta_\ell\asymp\min\\!\left\\{ \tau_{\max}^{-1/2}, \sqrt{\frac{\tau_{\max}}{\max\\{1,\tau_{\mathrm{avg}}\\}}}\right\\}.
FedAWE (Eq. (11))ηℓ≲δ,ηℓηg≲δ1−ρ1+ρ\displaystyle \eta_\ell\lesssim\delta,\qquad \eta_\ell\eta_g\lesssim \delta\frac{1-\sqrt{\rho}}{1+\sqrt{\rho}}.
MIFA (Theorem 6.1)η∝(1+νˉ)−1/2\eta\propto(1+\bar\nu)^{-1/2}.

For prior methods, ≍\asymp, ≲\lesssim, and ∝\propto suppress factors unrelated to participation. τmax⁡\tau_{\max} is FedSUM's maximum inactivity, and τavg\tau_{\mathrm{avg}} averages the per-round maximum inactivity over time; νˉ\bar\nu averages MIFA's per-client inactivity bounds. Λ\Lambda and FedAWE's mixing factor ρ\rho also depend on participation.

These dependencies enter sufficient convergence conditions; they do not prescribe the only empirically useful learning rates. In practice, we tune TRUSS over the fixed local and global learning-rate grids in Appendix B.3 (similar to other methods), without supplying δ\delta. This is empirical tuning, not a participation-independent theoretical step-size guarantee.

We also conducted CIFAR-10 experiments under exponential participation, varying γ\gamma to change δ\delta while keeping mean participation fixed. The table reports mean test accuracy over three seeds after 500 rounds, with rates tuned separately for each setting.

δ\deltaMean test accuracy (%)Tuned ηℓηg\eta_\ell\eta_g
0.1073.60.25
0.0573.10.10
0.0172.30.25

The tuned product is 0.25 at both δ=0.10\delta=0.10 and δ=0.01\delta=0.01: smaller δ\delta does not necessarily require a smaller tuned product in these experiments. Mean accuracy ranges from 72.372.3\\% to 73.673.6\\% after separate tuning.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer,

We wanted to briefly check if our new comparisons with FedSUM-CR, the expanded scaling experiments for clients and local steps, and the learning rate clarifications addressed your concerns. Please let us know if you need further details.

Best, Authors

Thank you for the detailed responses and additional experiments. The rebuttal has mostly addressed my concerns, particularly through the direct comparison with FedSUM-CR, the empirical analysis of communication-round scaling, and the clarification of learning-rate selection without participation information. Given that my initial assessment was already positive, I will maintain my current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.