- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
SeedFlood is a method that is used for decentralized zeroth-order fine tuning of LLMs. By leveraging the seed-reconstructible structure of zeroth-order updates in addition to flooding, the method combats the communication overhead and global consensus challenges. Each nodes update is sent as a scalar pair and flooded across the entire network, so all nodes apply identical updates and retain global consensus. To manage aggregation costs as the network scales, pertubations are restricted to a finite set of rank-one directions.
Critical strengths
[S1] The motivation is clearly explained (why seed-based methods don't fit gossip averaging well), the method is made very clear (figure 1).
[S2] The aggregation design shows real practical benefit for networking scaling, showing communication costs falling from O(nd) to O(n + rd), showing aggregation performance improvements of up to ~2000x.
[S3] The experiments cover a broad range decentralized methods (table 7), showing the benefits in reduced message sizes compared to zeroth-order gossip baselines.
[S4] The limitations section is honest, and delayed flooding has experimental support (appendix B.3)
Critical weaknesses
[W1] Exact consensus requires every message to reach every node but there is little discussion on lost messages, the straggler effect or faulty nodes which is a common failure mode in decentralized regimes (local compute connected by the internet). Mentioning these could make the suitability for decentralized training stronger
[W2] It could be useful to understand what the behaviour is like under realistic bandwidth and latency. Costs are currently reported in bytes. Given that flooding requires hops equal to the network diameter. This could inform of limits of the system.
[W3] DZSG, the zeroth-order gossip baseline is left out of the scaling experiments (figures 4, 5). It could useful to include this comparison since it will show the degradation of gossip as the network grows.
Questions for authors
Please address the weaknesses above.
2 comments
W3. Including the zeroth-order gossip baseline in the scaling experiments
Thank you for this constructive suggestion. We agree that including a zeroth-order gossip baseline in the scaling experiments would help isolate the benefits of flooding-based consensus from those of zeroth-order optimization itself.
Since DZSGD uses zeroth-order updates like SeedFlood but relies on gossip averaging, we expected it to suffer from topology-dependent consensus degradation and thus exhibit less competitive performance, particularly as the number of clients increases. This expectation was supported by our main experiments in Section 5.2, where DZSGD and its variants generally underperformed SeedFlood while incurring substantially higher communication costs. Based on these results, we considered DZSGD a less competitive baseline and prioritized stronger first-order PEFT baselines in the subsequent scaling experiments.
Nevertheless, we agree that directly evaluating DZSGD across increasing client counts would provide valuable evidence of how gossip-based consensus affects zeroth-order optimization at larger collaboration scales.
We are currently conducting these additional experiments. However, they are time-consuming primarily because zeroth-order methods require substantially larger iteration budgets than first-order methods, making evaluations across multiple client counts computationally expensive. It remains uncertain whether the experiments will finish within the rebuttal period. If the results become available in time, we will report them during the discussion. Otherwise, we will acknowledge this missing comparison as a limitation and seek to include it in the final version of the paper.
We appreciate the reviewer's suggestions, which help clarify SeedFlood's practical behavior and identify useful directions for further evaluation.
References
- Lian et al. (2018). [Asynchronous Decentralized Parallel Stochastic Gradient Descent] ICML 2018.
- Spiridonoff et al. (2020). [Robust Asynchronous Stochastic Gradient-Push: Asymptotically Optimal and Network-Independent Performance for Strongly Convex Functions] JMLR 21(58), 2020.
- Xiong et al. (2024). [Straggler-Resilient Decentralized Learning via Adaptive Asynchronous Updates] MobiHoc 2024.
Thank you for the thoughtful review and for recognizing the clear motivation, practical aggregation design, and breadth of our evaluation. We address the suggestions on stragglers and failures, realistic network conditions, and the zeroth-order gossip baseline below.
W1. Stragglers, lost messages, and faulty nodes
Thank you for this helpful suggestion. As with other distributed optimization algorithms, flooding cannot reliably exchange information with a device while that device is faulty or temporarily unavailable.
An important structural property of SeedFlood is that all missed weight updates can be represented as a set of compact seed-scalar pairs. This suggests a recovery mechanism in which, once a temporarily unavailable client returns, it receives the missed updates from its neighbors and reconstructs them to recover exact consensus with the rest of the network.
Gossip averaging does not directly reconstruct the individual weight changes missed during an interruption. The two mechanisms may therefore differ in both the communication required for recovery and the speed at which consensus is restored.
Prior work on decentralized learning has also studied mechanisms for reducing the effects of stragglers and communication failures, including asynchronous updates, adaptive selection of communication participants, and compensation for lost or delayed information through cumulative-message differences or step-size adjustments (Lian et al., 2018; Spiridonoff et al., 2020; Xiong et al., 2024). These mechanisms have often been studied as relatively independent algorithmic components, and some may also be applicable to SeedFlood.
However, establishing whether SeedFlood is more robust than gossip in practice would require controlled experiments and an understanding of realistic straggler and failure patterns. A comparison based only on the communication mechanisms would not be sufficient for a fair, comprehensive conclusion. We view this as a separate research direction that warrants a dedicated study beyond the scope of the present work.
We agree that discussing these possibilities and limitations will help readers better assess the method’s suitability for decentralized training, and we will add this discussion to the manuscript.
W2. Behavior under realistic bandwidth and latency
We agree with this suggestion. Reviewer A7bk also raised a related concern, and we have added estimated end-to-end wall-clock-time comparisons that account for both network conditions and the number of iterations required to reach a target validation loss. These results are presented in Appendix I (Figures 11 and 12), with representative network operating points summarized in Table 12. Please also see our response to Reviewer A7bk.
As the reviewer notes, SeedFlood's communication time depends on both network diameter and per-hop latency. Its compact messages make it particularly attractive in bandwidth-constrained settings, but high latency can increase the cost of multi-hop propagation.
Our wall-clock analysis incorporates measured per-iteration computation time and a communication-time model based on bandwidth, latency, and the number of communication rounds (Appendix I, Equations 78 and 79).
Importantly, the end-to-end comparison can be more favorable than the per-round communication cost alone suggests. Figure 11 shows that, as the number of clients increases, SeedFlood requires fewer iterations to reach the target loss, while gossip-based baselines increasingly suffer from slower convergence or fail to reach the target within the iteration budget.
At 16 clients, the DSGD-FFALoRA baseline is faster at the evaluated network operating points. However, at larger collaboration scales, SeedFlood becomes faster under many practical network conditions. In particular, at 128 clients on SQuAD, SeedFlood is faster at all nine evaluated operating points, using the relaxed target-loss threshold specified in Appendix I for comparisons where the baseline does not reach the original target.
Figure 12 further extends the comparison to Qwen3-8B instruction tuning. SeedFlood achieves substantial wall-clock advantages over full-parameter DSGD in bandwidth-constrained environments, while remaining competitive with DSGD-FFALoRA across most evaluated network conditions.
We also investigate delayed flooding in Appendix B.3. By limiting the number of flooding hops per training iteration, delayed flooding relaxes the requirement for full propagation at every step. Our preliminary experiments show that moderate propagation delays can preserve performance comparable to full flooding, suggesting a possible direction for mitigating latency overhead.
Overall, these results clarify both the advantages and limitations of SeedFlood: its reduced communication volume and improved convergence scaling can translate into faster end-to-end training, but sufficiently high multi-hop latency remains an important limitation.