- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies federated training of selective SSMs. It derives architecture-aware gradient and smoothness bounds, and then uses them to obtain convergence guarantees for FedAvg and FedProx. The theory is further validated with synthetic experiments and practical Mamba2 experiments.
Critical strengths
- Architecture-aware analysis. Instead of assuming bounded gradients, the paper derives the gradient and smoothness bounds directly from the selective SSM structure, which makes the dependence on recurrent stability and model parameters explicit.
- Meaningful connection to federated optimization. The paper further incorporates these SSM-specific quantities into the convergence analysis of FedAvg and FedProx, providing insight into how recurrent stability, local updates, and client heterogeneity affect federated training.
Critical weaknesses
- Limited to Full Client Participation. The analysis assumes that all clients participate in every communication round and does not study partial participation. As partial client participation is common in federated learning, this limits the practical applicability of the theoretical results.
- Strong stability assumption. From an FL perspective, assuming that all local iterates remain in a stable parameter region seems fairly strong, especially with heterogeneous data and multiple local steps. This concern is even stronger in the multi-layer setting, where the stability and boundedness conditions must hold simultaneously across all layers throughout training. This condition is enforced by projection in the synthetic experiments, but it is unclear whether it holds in the practical Mamba2 setting. I am not an expert in selective state space models, so further clarification on this assumption would be helpful.
- Limited characterization of client heterogeneity. Sequence-level TV may saturate under strong heterogeneity and therefore provide limited resolution between different non-IID regimes. Also, the heterogeneity term disappears when . Although this follows from the full-participation setting, it is less representative of practical FL with partial participation, where heterogeneity can still matter through client sampling. The authors should clarify this limitation.
Questions for authors
- See weakness.
- Could the authors extend the analysis to SCAFFOLD and provide a direct comparison among the convergence bounds of FedAvg, FedProx, and SCAFFOLD for selective SSM training? It would be particularly interesting to understand how the architecture-specific factors identified in this paper, such as recurrent stability, parameter envelopes, and client drift, affect the relative tightness of the three bounds.
- The synthetic experiments study the effect of extensively, but the practical Mamba2 experiments do not appear to report the realized layerwise stability factors. Could the authors measure these quantities during Mamba2 training and examine whether the theoretical dependence on recurrent stability is also observed in practice?
- I am not an expert in selective state space models, so I would defer to the opinions of other reviewers.
- I am open to raising my score if the authors adequately address my concerns in the rebuttal.
9 comments
Response (2/4)
Stability assumption. In standard Mamba2, the transition parameters and discretization steps are parameterized as and , respectively. As such, as long as and remain uniformly bounded over the considered training neighborhoods (i.e. do not diverge to ), we can take their infima and choose: and resulting in . For an appropriate choice of , the bounded neighborhood together with bounded layer inputs ensure such finite lower bounds. The same reasoning applies separately to every layer, allowing layer-specific margins.
Containment of the local iterates does not depend on data heterogeneity. By Theorem 4.4 and Lemma B.2, every per-sequence gradient in the neighborhood has norm at most , so local SGD steps displace a client by at most , and containment in holds whenever . Under Mamba2's parameterization every finite parameter value gives , so local updates cannot leave the stable region. The assumption therefore reduces to the boundedness discussed above.
In the synthetic experiments, projection imposes a specific stability upper bound, such as , enabling controlled comparisons across different stability regimes. This is stronger than requiring that some exists: projection lets us select the stability ceiling in advance instead of obtaining it from the parameter and input bounds. The additional Mamba2 results reported below show that a single bound holds in practice during federated training, supporting the practicality of this assumption.
Response (3/4)
SCAFFOLD analysis. Our architecture-specific gradient and smoothness bounds provide the results needed to analyze SGD-based SCAFFOLD. The additional step is to track the evolution of its control variates along with the optimization error. Below we outline this extension, and will include the full rigorous analysis in the revised manuscript.
The SCAFFOLD local update is where , and estimates the gradient of the local objective ( denotes the mini-batch gradient of Assumption 4.6). All assumptions, common parameter envelopes, and stable-neighborhood containment conditions of Theorem B.5 hold. In particular, the corrected local iterates, including their endpoints, lie in . This holds whenever , since the control variates are averages of gradients bounded by . In this analysis, we use , , , and .
We use the standard control-variate update, , the aggregation rule and . We further define C_r := \sum_kp_k\\,\mathbb{E} \left\\| (c_k^r-c^r) - \bigl(\nabla\mathcal{L}_k(W^r)-\nabla f(W^r)\bigr)\right\\|_F^2.
The main new step of the proof is using the -smoothness to bound local drift in terms of , the global gradient norm, and . By expanding the local displacement and applying smoothness and the variance bounds from the paper, for , it follows that . Then, the control-variate update, together with this result and the cancellation of the corrections in the server aggregate, yields . By the smoothness of and conditional unbiasedness, it follows that .
Substituting the local drift bound into the descent inequality and adding the control-error recursion scaled by an appropriate constant, under , it follows that: . Summing over , telescoping the terms, using , , and dividing by , we get: . For zero initial controls, Lemma B.4 gives , so the heterogeneity contribution enters through an initialization term that decays as , while the architecture-specific dependence remains through and .
Below we compare FedAvg, FedProx, and SCAFFOLD, where and are evaluated on the no-larger neighborhood :
| Method | Local-noise drift term | Heterogeneity term |
|---|---|---|
| FedAvg | ||
| FedProx | ||
| SCAFFOLD |
At equal , , and , and for , the SCAFFOLD bound is smaller than the FedAvg bound exactly when , which is approximately for large , i.e. when heterogeneity dominates local sampling noise. The remaining two terms of the FedAvg and SCAFFOLD bounds coincide. For nearly IID data it is larger, and SCAFFOLD also requires the stricter step-size condition . This is consistent with the SCAFFOLD results in Table 2. The architecture-specific factors enter both terms through , so they do not change which bound is smaller but amplify the gap as . FedProx is the only one of the three that reduces the architecture-specific constants themselves, through the tighter envelopes on , at the cost of in .
Response (1/4)
We thank the reviewer for the insightful feedback. Below, we address the raised concerns and questions.
Full client participation and limited client heterogeneity analysis. The main objective of this work is to establish a theoretical foundation for architecture-aware distributed optimization of selective SSMs. Full client participation is a well-defined federated learning setting that allows us to isolate the effects of recurrent stability, parameter envelopes, and local client drift. Given this setting and the stated assumptions, we derive sequence-length-independent gradient bounds and local smoothness estimates for single-layer and multi-layer selective SSMs, and establish how recurrent stability and parameter envelopes enter FedAvg and FedProx convergence guarantees, including the trade-off between proximal radius contraction and the additional smoothness cost.
We agree that partial participation and a more detailed characterization of heterogeneity are important research directions. A comprehensive analysis would involve accounting for client-sampling variability and characterizing how these quantities interact with the architecture-specific sensitivities and local-update dynamics of selective SSMs. This question already substantially expands the theoretical scope and could form the basis of a dedicated follow-up study, however it is beyond the scope of a work meant to establish foundations for such analysis. Nevertheless, we agree that these limitations and their implications should be made explicit. We therefore provide an illustrative discussion below.
Consider partial participation with arbitrary client weights pk≥0 (∑kpk=1) and a single local SGD step (E=1). At each round r, let Cr be a multiset of m=∣Cr∣ independent client draws with replacement, where each draw selects client k with probability pk (and each client draw uses an independent mini-batch). Define D_r:=\sum_{k=1}^{K}p_k \left\\|\nabla \mathcal L_k(W^r)-\nabla f(W^r)\right\\|_F^2 and gr:=m1∑k∈Crgk,Ξ(Wr). Let Er denote expectation conditional on the training history before round-r sampling. Under the stated assumptions, the fact that Er[gr]=∇f(Wr), and by Lemmas B.3 and B.4, it follows that: \mathbb E_r \left\\|\widehat g_r - \nabla f(W^r)\right\\|_F^2 \le \underbrace{\frac{G_{\mathrm{SSM},R}^{2}}{m\Xi}}_{\text{within-client noise}} + \underbrace{\frac{D_r}{m}}_{\text{client-selection noise}} \le \frac{G_{\mathrm{SSM},R}^{2}}{m}\left(\frac1\Xi+4 \mathcal H_{\mathrm{TV}}\right).
For E=1 and η≤1/LSSM,R, this gives T1∑r=0T−1E∣∇f(Wr)∣F2≤ηT2(f(W0)−f∗)+mηLSSM,RGSSM,R2(Ξ1+4HTV). Thus, as the reviewer pointed out, under client sampling heterogeneity enters even at E=1 through the client-selection term, whereas it vanishes at E=1 under full participation. Extending this to multiple local updates, and refining the worst-case heterogeneity bound with client-dependent gradient sensitivities and stability margins, is left for future work.
Regarding HTV, we agree that sequence-level total variation can saturate. We note this limitation in Section 5 and quantify it in our synthetic experiments (Appendix H.1, Figure 8b): the sequence-level squared-TV aggregate is 0.56 at K=4 and is unchanged by the partition, whereas a token-level analogue varies by two orders of magnitude. A token-level or moment-based dissimilarity would give finer resolution but requires a separate bound. We will include the partial client participation discussion along with the limitation of the existing analysis as a new section in the Appendix of the revised manuscript.