acceptodds
Under review as a conference paper at ICLR 2027

SeedFlood: A Step Toward Scalable Decentralized Fine-Tuning of LLMs

Abstract

This work presents SeedFlood, a new approach to decentralized LLM fine-tuning designed to scale across large models, large collaborations, and complex network topologies while achieving global consensus with negligible communication overhead. Traditional methods suffer from high communication costs that grow with model size, while information decay over network hops renders global consensus inefficient. SeedFlood takes a significant departure from these practices by exploiting the seed-reconstructible structure of zeroth-order gradients and effectively making the messages to transmit near-zero in size, allowing them to be flooded to every client in the network, and thereby enhancing scalability of decentralized training. Consequently, SeedFlood enables training in regimes previously considered impractical, such as billion-parameter scale models or distributed across hundred of clients. Despite the intrinsic optimization inefficiency of zeroth-order estimation, SeedFlood can match or surpass first-order gossip baselines at larger collaboration scales by avoiding consensus errors, while requiring orders-of-magnitude less communication. We also provide theoretical analysis to formalize that SeedFlood avoids topology-dependent consensus terms in the convergence bound while retaining the acceleration enabled by increased client participation.

Primary area:
optimization
Keywords:
Decentralized training, Collaborative training, Zeroth order optimization, LLM fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_8FXkOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

SeedFlood is a method that is used for decentralized zeroth-order fine tuning of LLMs. By leveraging the seed-reconstructible structure of zeroth-order updates in addition to flooding, the method combats the communication overhead and global consensus challenges. Each nodes update is sent as a scalar pair and flooded across the entire network, so all nodes apply identical updates and retain global consensus. To manage aggregation costs as the network scales, pertubations are restricted to a finite set of rank-one directions.

Critical strengths

[S1] The motivation is clearly explained (why seed-based methods don't fit gossip averaging well), the method is made very clear (figure 1).

[S2] The aggregation design shows real practical benefit for networking scaling, showing communication costs falling from O(nd) to O(n + rd), showing aggregation performance improvements of up to ~2000x.

[S3] The experiments cover a broad range decentralized methods (table 7), showing the benefits in reduced message sizes compared to zeroth-order gossip baselines.

[S4] The limitations section is honest, and delayed flooding has experimental support (appendix B.3)

Critical weaknesses

[W1] Exact consensus requires every message to reach every node but there is little discussion on lost messages, the straggler effect or faulty nodes which is a common failure mode in decentralized regimes (local compute connected by the internet). Mentioning these could make the suitability for decentralized training stronger

[W2] It could be useful to understand what the behaviour is like under realistic bandwidth and latency. Costs are currently reported in bytes. Given that flooding requires hops equal to the network diameter. This could inform of limits of the system.

[W3] DZSG, the zeroth-order gossip baseline is left out of the scaling experiments (figures 4, 5). It could useful to include this comparison since it will show the degradation of gossip as the network grows.

Questions for authors

Please address the weaknesses above.

2 comments

Thank you for the thoughtful review and for recognizing the clear motivation, practical aggregation design, and breadth of our evaluation. We address the suggestions on stragglers and failures, realistic network conditions, and the zeroth-order gossip baseline below.


W1. Stragglers, lost messages, and faulty nodes

Thank you for this helpful suggestion. As with other distributed optimization algorithms, flooding cannot reliably exchange information with a device while that device is faulty or temporarily unavailable.

An important structural property of SeedFlood is that all missed weight updates can be represented as a set of compact seed-scalar pairs. This suggests a recovery mechanism in which, once a temporarily unavailable client returns, it receives the missed updates from its neighbors and reconstructs them to recover exact consensus with the rest of the network.

Gossip averaging does not directly reconstruct the individual weight changes missed during an interruption. The two mechanisms may therefore differ in both the communication required for recovery and the speed at which consensus is restored.

Prior work on decentralized learning has also studied mechanisms for reducing the effects of stragglers and communication failures, including asynchronous updates, adaptive selection of communication participants, and compensation for lost or delayed information through cumulative-message differences or step-size adjustments (Lian et al., 2018; Spiridonoff et al., 2020; Xiong et al., 2024). These mechanisms have often been studied as relatively independent algorithmic components, and some may also be applicable to SeedFlood.

However, establishing whether SeedFlood is more robust than gossip in practice would require controlled experiments and an understanding of realistic straggler and failure patterns. A comparison based only on the communication mechanisms would not be sufficient for a fair, comprehensive conclusion. We view this as a separate research direction that warrants a dedicated study beyond the scope of the present work.

We agree that discussing these possibilities and limitations will help readers better assess the method’s suitability for decentralized training, and we will add this discussion to the manuscript.


W2. Behavior under realistic bandwidth and latency

We agree with this suggestion. Reviewer A7bk also raised a related concern, and we have added estimated end-to-end wall-clock-time comparisons that account for both network conditions and the number of iterations required to reach a target validation loss. These results are presented in Appendix I (Figures 11 and 12), with representative network operating points summarized in Table 12. Please also see our response to Reviewer A7bk.

As the reviewer notes, SeedFlood's communication time depends on both network diameter and per-hop latency. Its compact messages make it particularly attractive in bandwidth-constrained settings, but high latency can increase the cost of multi-hop propagation.

Our wall-clock analysis incorporates measured per-iteration computation time and a communication-time model based on bandwidth, latency, and the number of communication rounds (Appendix I, Equations 78 and 79).

Importantly, the end-to-end comparison can be more favorable than the per-round communication cost alone suggests. Figure 11 shows that, as the number of clients increases, SeedFlood requires fewer iterations to reach the target loss, while gossip-based baselines increasingly suffer from slower convergence or fail to reach the target within the iteration budget.

At 16 clients, the DSGD-FFALoRA baseline is faster at the evaluated network operating points. However, at larger collaboration scales, SeedFlood becomes faster under many practical network conditions. In particular, at 128 clients on SQuAD, SeedFlood is faster at all nine evaluated operating points, using the relaxed target-loss threshold specified in Appendix I for comparisons where the baseline does not reach the original target.

Figure 12 further extends the comparison to Qwen3-8B instruction tuning. SeedFlood achieves substantial wall-clock advantages over full-parameter DSGD in bandwidth-constrained environments, while remaining competitive with DSGD-FFALoRA across most evaluated network conditions.

We also investigate delayed flooding in Appendix B.3. By limiting the number of flooding hops per training iteration, delayed flooding relaxes the requirement for full propagation at every step. Our preliminary experiments show that moderate propagation delays can preserve performance comparable to full flooding, suggesting a possible direction for mitigating latency overhead.

Overall, these results clarify both the advantages and limitations of SeedFlood: its reduced communication volume and improved convergence scaling can translate into faster end-to-end training, but sufficiently high multi-hop latency remains an important limitation.


W3. Including the zeroth-order gossip baseline in the scaling experiments

Thank you for this constructive suggestion. We agree that including a zeroth-order gossip baseline in the scaling experiments would help isolate the benefits of flooding-based consensus from those of zeroth-order optimization itself.

Since DZSGD uses zeroth-order updates like SeedFlood but relies on gossip averaging, we expected it to suffer from topology-dependent consensus degradation and thus exhibit less competitive performance, particularly as the number of clients increases. This expectation was supported by our main experiments in Section 5.2, where DZSGD and its variants generally underperformed SeedFlood while incurring substantially higher communication costs. Based on these results, we considered DZSGD a less competitive baseline and prioritized stronger first-order PEFT baselines in the subsequent scaling experiments.

Nevertheless, we agree that directly evaluating DZSGD across increasing client counts would provide valuable evidence of how gossip-based consensus affects zeroth-order optimization at larger collaboration scales.

We are currently conducting these additional experiments. However, they are time-consuming primarily because zeroth-order methods require substantially larger iteration budgets than first-order methods, making evaluations across multiple client counts computationally expensive. It remains uncertain whether the experiments will finish within the rebuttal period. If the results become available in time, we will report them during the discussion. Otherwise, we will acknowledge this missing comparison as a limitation and seek to include it in the final version of the paper.


We appreciate the reviewer's suggestions, which help clarify SeedFlood's practical behavior and identify useful directions for further evaluation.

References

  • Lian et al. (2018). [Asynchronous Decentralized Parallel Stochastic Gradient Descent] ICML 2018.
  • Spiridonoff et al. (2020). [Robust Asynchronous Stochastic Gradient-Push: Asymptotically Optimal and Network-Independent Performance for Strongly Convex Functions] JMLR 21(58), 2020.
  • Xiong et al. (2024). [Straggler-Resilient Decentralized Learning via Adaptive Asynchronous Updates] MobiHoc 2024.
Reviewer_A7bkOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents SeedFlood, a framework for scalable decentralized LLM fine-tuning that addresses connected bottlenecks. There are three main contributions: First, it represents zeroth-order updates with a random seed and a scalar coefficient, which removes model dimensionality as the dominant communication bottleneck in centralized federated learning. Second, it floods these compact messages throughout the network, allowing all clients to apply the same aggregated update and achieve exact consensus after complete propagation unlike the hop-wise dilution of gossip. Third, the work can avoid the cost of reconstructing high-dimensional update i.e the linear O(nd)O(nd) scaling, it uses periodically refreshed shared low-rank subspaces to aggregate updates in coefficient space before applying them to the model and the new aggregation cost is O(n+rd)O(n+rd).

The theoretical analysis establishes convergence under SeedFlood framework, and experiments on language models also demonstrate substantial communication savings with competitive task performance.

Critical strengths
  1. The framework is well-motivated: SeedFlood address the bottlenecks from large update messages, slow consensus and high update cost. This work made a meaningful contribution to communication-efficient decentralization fine-tuning.

  2. The empirical results can support communication efficiency and scalability. SeedFlood requires approximately 400 KB per edge over training and retains competitive task performance compared with full-parameter DSGD. Separate experiments scaling from 16 to 128 clients show stable or improving performance, while several gossip-based baselines deteriorate.

Critical weaknesses
  1. The evidence of practical training speed is a little bit limited: The paper didn't show that the communication saving makes training faster in practice. In a real communication network, SeedFlood may need more training steps that first-order methods, but the experiences only simulate many clients on a few GPUs.
Questions for authors

I would like to know: Considering the additional training steps and multi-hop communication, under what kinds of network would SeedFlood reach the same task performance faster than the main baselines?

1 comment

We greatly appreciate the reviewer’s encouraging assessment of SeedFlood’s motivation, communication efficiency, and scalability. Thank you also for highlighting the need to establish whether these savings translate into faster end-to-end training. We address this concern and the question about network conditions together below.


Practical training speed under different network conditions

We fully agree that the number of training steps and the cost of multi-hop communication should be considered together. To address this, we compare the estimated end-to-end wall-clock time of SeedFlood and the baselines across network conditions with different bandwidths and latencies, using the SST-2 and SQuAD scaling experiments (Section 5.4) and the larger-model experiment (Appendix B.5). The corresponding wall-clock-time analyses and figures are provided in Appendix I (Figures 11 and 12).

For each method, we use the number of iterations required to reach the target validation performance and account for the communication rounds and payload sizes under each simulated network condition. Thus, these comparisons incorporate the training steps needed to reach the target, rather than comparing only communication volume or per-step cost.

In the plots, blue regions indicate network conditions where SeedFlood reaches the target faster than the comparison method, and red regions indicate the opposite. The markers represent real-world network operating points based on reported measurements: circles for cloud networks, squares for fixed broadband and edge-to-cloud WAN links, and diamonds for satellite networks.

SST-2 and SQuAD scaling experiments

Figure 11 in Appendix I presents the estimated wall-clock-time comparisons for the SST-2 and SQuAD scaling experiments, based on the iterations-to-target results in Section 5.4 under the experimental setup of Section 5.3.

As the number of participating clients increases, SeedFlood converges in fewer iterations, eventually reversing the iteration-count disadvantage relative to the baselines and reaching the target faster.

Because SeedFlood propagates small messages over multiple hops, high latency can increase its per-step communication cost. Nevertheless, its improved convergence with more clients makes it increasingly favorable as the collaboration grows.

In these small-model experiments, computation and the required number of iterations are major contributors to end-to-end training time under low-latency conditions, whereas multi-hop communication latency becomes increasingly important in high-latency regimes. To better examine the wall-clock benefit of reduced communication at a larger model scale, we provide the same comparison for the 8B-model experiment in Figure 12 of Appendix I.

Qwen 8B instruction tuning on Tulu-IF

Figure 12 in Appendix I presents the corresponding wall-clock-time comparisons for Qwen3-8B instruction tuning on Tulu-IF, based on the experiment described in Appendix B.5.

Against the full-parameter DSGD baseline, SeedFlood has a substantial advantage across most network conditions, even after accounting for the total number of training steps needed to reach the same loss.

Against the DSGD-FFALoRA baseline, SeedFlood is slightly faster over most of the evaluated network conditions, with the exception of regimes with extremely high latency.

Together, these results clarify when the communication savings translate into faster end-to-end training. SeedFlood benefits from larger collaborations and bandwidth-constrained settings, while very high per-hop latency can reduce its advantage. The larger-model experiment also shows that the advantage can persist after accounting for differences in the number of training steps.

We appreciate this constructive suggestion. Including the required training steps in the wall-clock-time comparison provides a more complete view of how SeedFlood addresses the practical constraints of decentralized fine-tuning.

Reviewer_XKHaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SeedFlood, a decentralized framework for communication-efficient LLM fine-tuning. SeedFlood represents each zeroth-order update using a random seed and a scalar coefficient, allowing the update to be reconstructed without transmitting model-sized gradients. These compact messages are flooded across the network so that all clients apply the same globally aggregated update. To reduce the computation required to reconstruct updates from many clients, the method further restricts perturbations to a periodically refreshed low-rank subspace and performs aggregation in its coefficient space. The paper provides convergence analysis under exact flooding and evaluates the method on multiple language tasks, model scales, network topologies, and up to 128 clients.

Critical strengths
  1. The paper addresses an important problem in decentralized LLM fine-tuning. Combining seed-reconstructible zeroth-order updates with flooding is a natural and practically meaningful way to reduce model-size-dependent communication while avoiding the consensus degradation of gossip averaging.

  2. SeedFlood demonstrates a substantial communication advantage. In the main experiments, it reduces the transmitted volume by several orders of magnitude relative to dense first-order and zeroth-order baselines, while maintaining competitive task performance. The experiments also cover different model families and collaboration sizes, showing the practical potential of the proposed framework.

Critical weaknesses
  1. The main scalability experiments use only 1,024 training examples per task. With 16 clients, each client has only 64 examples; if the batch size of 16 is local, 5,000 iterations correspond to approximately 1,250 local data passes. The paper nevertheless claims convergence “without overfitting” without reporting learning curves or selected checkpoint iterations. Moreover, because the per-client batch size is fixed, increasing the number of clients also increases the total samples and function evaluations per iteration. Figures 4 and 5 therefore confound client participation with additional computation. Appendix B.5 provides one useful larger-data experiment, but only in a single setting.

  2. First-order baselines are trained for 500 iterations, whereas zeroth-order methods receive 5,000. Different budgets may be appropriate if convergence is demonstrated, but the paper does not provide sufficient curves or compute-matched results. Consequently, the comparison among zeroth-order methods is more convincing than the claimed comparison with first-order training.

Questions for authors
  1. Is the batch size in Table 6 local or global, and how are batches sampled when a client has fewer examples than the batch size? Please report the selected checkpoint iterations and the corresponding training and validation curves.

  2. How does SeedFlood perform at 500 iterations, and do the first-order baselines improve with a longer budget and early stopping? Results matched by iterations, processed samples, or compute would clarify the comparison.

  3. What are the main technical differences between the convergence analysis and standard data-parallel zeroth-order analysis? In particular, how should the 1/n1/n improvements be interpreted when the total dataset is fixed but the total batch size and computation per iteration increase with nn?

5 comments

We thank the reviewer for the careful evaluation and constructive feedback. We appreciate the recognition of SeedFlood’s practical potential and communication efficiency, and address the concerns about experimental budgets, client scaling, and convergence analysis below.


W1. Evaluation configuration in the scalability experiments

Thank you for pointing out the ambiguity in our description. In our initial experiments, we allocated 5,000 iterations to all methods for a fair comparison. However, first-order methods generally achieved their best validation checkpoints within 500 iterations. Counting the communication over all 5,000 iterations would therefore overestimate the communication budget required by these methods. We accordingly used budgets of 500 iterations for first-order methods and 5,000 iterations for zeroth-order methods, with hyperparameter tuning for the respective budgets.

The phrase “sufficient for convergence without overfitting” was intended to explain that the budgets were sufficient for convergence while limiting unnecessary iterations after overfitting. It was not intended to claim that every method converged at exactly 500 or 5,000 iterations without any overfitting. We will revise this sentence to make the intended meaning clear. We have added the selected checkpoint iterations for the experiments in Sections 5.2 and 5.3 in Tables 9 and 10 of Appendix H.

Regarding the additional computation from more clients, the reviewer’s interpretation is correct. With a fixed local batch size, increasing the number of participating clients increases the total computation per iteration linearly. The resulting improvement in convergence is the familiar benefit of data-parallel training.

The key distinction is that conventional decentralized baselines cannot fully benefit from this additional computation in large networks because limited communication prevents the models from maintaining consensus. By maintaining exact consensus after complete flooding, SeedFlood can make effective use of the additional computation contributed by more clients.

Our scaling experiments do not consider splitting a fixed total computation budget across more clients. We will clarify that the gains come from SeedFlood’s ability to use the additional computational resources effectively, rather than from increasing the client count while holding the total computation fixed.


W2 / Q2. First-order versus zeroth-order budgets and an iteration-matched comparison

We address these related comments together. To clarify the choice of training budgets, we report the selected checkpoint iterations in Appendix H. First-order methods generally reached their best validation checkpoints within 500 iterations in our initial experiments.

We additionally report an iteration-budget-matched comparison between SeedFlood and DSGD-FFA, the strongest first-order baselin for the scaling experiments in Section 5.3. The results are provided in Table 11 of Appendix H, where DSGD-FFA is evaluated at its best-validation checkpoint within 500 iterations and SeedFlood at exactly iteration 500.

SeedFlood is generally weaker at this early checkpoint in small networks, while its relative performance improves as the network grows. For example, at 128 clients it exceeds DSGD-FFA on SST-2, BoolQ, and SQuAD, although not on RTE. This is consistent with our main observation that SeedFlood better retains the benefits of collaboration in larger networks.

These SeedFlood results are taken directly from the 500-iteration checkpoints of the 5,000-iteration runs, without additional hyperparameter tuning for the shorter budget. They may therefore underestimate its performance when tuned specifically for 500 iterations.

Since first-order baselines generally reached their best validation checkpoints before 500 iterations in the initial experiments, simply extending the budget is unlikely to produce a dramatic improvement in those settings. Section 5.4 also provides a complementary comparison using a broader and finer learning-rate grid and a common target validation loss. In large networks, the first-order baselines either fail to reach this target within 10,000 iterations or require very small learning rates and more iterations than SeedFlood.

In small networks, there is a trade-off between convergence speed and communication cost; in large networks, SeedFlood becomes more favorable. Because the methods differ in both convergence speed and per-step cost, matching iteration counts alone does not capture the full trade-off.

To provide an end-to-end comparison, we have reorganized the experiments from Section 5.4 and Appendix B.5 into wall-clock-time comparisons in Appendix I. Figures 11 and 12 show the network conditions under which SeedFlood reaches the target loss faster than the baselines. These results demonstrate that SeedFlood can achieve faster convergence in wall-clock time than first-order methods when many clients participate.


Q1. Local batch size and checkpoint selection

The batch size in Table 6 is local. For example, a batch size of 16 means that each client computes or estimates a gradient using 16 samples. When the number of samples available to a client is smaller than the configured batch size, we use the smaller of the two values as the effective local batch size.

We have added the selected checkpoint iterations for all experiments in Sections 5.2 and 5.3 to Tables 9 and 10 in Appendix H.


Q3. Relationship to standard data-parallel zeroth-order analysis

Our convergence analysis differs from standard data-parallel zeroth-order analysis in its treatment of the low-rank-factorized gradient estimator. This analysis characterizes how the estimator’s hyperparameters affect convergence. The remaining aspects intentionally align with standard data-parallel zeroth-order analysis.

SeedFlood’s contribution is to make a decentralized system behave equivalently to centralized data-parallel training through its algorithmic design. This equivalence comes from redesigning the communication and aggregation mechanism, rather than from a new analytical technique for handling consensus error.

Accordingly, when the total dataset is fixed but the total batch size and computation per iteration increase with the number of clients, the improvement should be interpreted as the benefit of additional computation, as discussed in our response to W1. SeedFlood allows this benefit to be retained in a decentralized network by maintaining exact consensus.


We appreciate the reviewer’s suggestions, which help us clarify both the scope of the scaling results and the trade-offs in the comparison with first-order methods. We hope that these clarifications and additional results address the concerns raised. If so, we would be grateful if the reviewer would reconsider the initial assessment.

Thank you for the response and revision. The added checkpoint results, 500-iteration comparison, larger-data experiment, and wall-clock analysis are useful. The theoretical clarification also explains that the 1/n1/n improvement comes from averaging more zeroth-order estimators, while flooding preserves exact consensus.

My main empirical concerns, however, are only partially resolved. The primary experiments still use only 1,024 examples per task; with 16 clients, a local batch size of 16, and 5,000 iterations, this amounts to approximately 1,250 local passes over each client’s data. Several best checkpoints occur near or at the final iteration, and no learning curves are provided, so the remaining claim of convergence “without overfitting” is not adequately supported. The larger-data experiment is helpful but limited to one setting. Under 500 iterations, SeedFlood outperforms the first-order baseline in only 4 of 16 settings, and the comparison is neither sample- nor compute-matched. Moreover, increasing the number of clients also increases total computation, so the scaling results do not establish an advantage under a fixed budget. I recognize SeedFlood’s communication advantage, but these limitations continue to restrict its main empirical claims. I therefore maintain my current recommendation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.