- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
DECA proposes a decentralized large model fine-tuning framework for non IID data, which reduces video memory and computational overhead by updating model parameters block by block, and utilizes consensus information from neighboring models to correct Adam's first-order and second-order momentum estimates, enabling multiple clients to collaboratively fine tune the main parameters of the large language model without a central server.The paper further presents a non-convex convergence analysis and evaluates the method on classification and generation tasks using Qwen and Llama models ranging from 1.5B to 8B parameters.
Critical strengths
-
Enabling decentralized fine-tuning of large language models without a central parameter server is an important research direction. The combination of limited training resources and non-IID client data presents meaningful challenges beyond conventional centralized fine-tuning.
-
The proposed framework combines memory-efficient block-wise optimization with peer-to-peer model aggregation. This design provides a plausible approach to updating a large portion of the model parameters while reducing the optimizer-state memory footprint.
-
The method achieves the best average classification accuracy and F1 score among the evaluated decentralized baselines on both Qwen2.5-3B and Llama-3.1-8B. The stronger non-IID experiment also have good results.
-
The paper includes classification and generation tasks, multiple communication topologies, different client counts, and block-granularity studies. The measured memory reduction with finer parameter partitioning is consistent with the motivation for block-wise optimization.
Critical weaknesses
-
The ultimate optimization objective of Formula 1 and Algorithm 1 is the arithmetic mean of all client models. But this global model is not seen in the main methods of this article or in the specific content of Algorithm 1. Although DECA performs neighbor based aggregation during training, each client maintains its own model parameters and optimizer state. In non IID data, the client model does not need to be identical after a limited number of communication rounds. If the final global model is explicitly constructed through additional aggregation processes, the author should explain how this operation can be achieved without central coordination, and whether its communication costs are included in the efficiency analysis
-
The paper appropriately cites BAdam and Quasi-Global Momentum (QGM) as related work. However, the relationship with QG-DAdam deserves a more direct discussion.Lin et al. (ICML 2021) have explored quasi global momentum in decentralized adaptive optimization and released some variants. DECA differs in its specific two-stage combination of local gradients and consensus discrepancies, as well as its integration with sequential block-wise training.The comparison with Dec-LoRA and DeCAF also changes the trainable parameterization, making it difficult to isolate the optimizer contribution.A direct comparison between DECA and block-wise QG-DAdam, under identical model partitioning, training budgets, and communication conditions, would substantially strengthen the novelty claim.
-
The resource-efficiency evaluation primarily reports communication volume and computational latency per training step. However, these measurements do not establish the total communication and computation required to complete decentralized fine-tuning. According to Algorithm 1, each of the B parameter blocks undergoes R inner updates and communication operations in each of T global rounds. Thus, the number of block-level communication events per client is TBR. Although communicating one block reduces the payload relative to full-model transmission at an individual step, this reduction must be evaluated together with the number of communication events. Table 8 shows that DECA actually transmits more data per reported communication step than Dec-LoRA and DeCAF. Therefore, the current evidence does not demonstrate lower cumulative communication overhead relative to the main decentralized PEFT baselines. The authors should report the total number of communication events, cumulative transmitted bytes, total training FLOPs, and end-to-end wall-clock time under controlled training budgets.
-
Section 5.1 states that every two consecutive Transformer layers form a trainable block and reports 32, 36, 28, and 32 blocks for Llama-2-7B, Llama-3.1-8B, Qwen2-1.5B, and Qwen2.5-3B, respectively. But, the Llama-3.1-8B results in Table 1 (78.91% average accuracy and 72.12% average F1) exactly match the granularity-1 results in Table 7, whereas the granularity-2 results are different.This raises uncertainty regarding the actual configuration used in the main experiments.
Questions for authors
1.BMA mechanism: What are the independent contributions of the first- and second-moment corrections? Can the authors provide separate ablations and directly measure client drift or consensus error? 2.For h, the normalization in the text is actually a unit normalization, does it ignore the amplitude difference? Is it better to normalize all h together
2 comments
We appreciate the reviewer for these insightful comments. Follows are the second part of our response to the main concerns of the reviewer.
Total Communication and Computation
In our experiments, we ensure that all baseline methods have the same number of updates and communication rounds, resulting in an identical total number of communication events. The table below reports the total number of communication events, cumulative bytes transmitted, total training FLOPs, and actual end-to-end wall-clock time for both Dec-LoRA and DECA across four model-task configurations. (Specifically, the sequence length is set to 1024 for Qwen2-1.5B and Qwen2.5-3B, and 512 for Llama-2-7B and Llama-3.1-8B; all other training-related parameters remain consistent.) As shown, although DECA incurs a higher cumulative communication overhead than Dec-LoRA, it outperforms Dec-LoRA in terms of both total training FLOPs and actual end-to-end wall-clock time.
| Model | Methods | Total communication events | Cumulative transmitted (TiB) | Total training (PFLOPs) | End-to-end wall-clock time (s) |
|---|---|---|---|---|---|
| Qwen2-1.5B | DECA | 5376 | 18.09 | 938.806 | 3605.6 |
| Qwen2-1.5B | Dec-LoRA | 5376 | 15.074 | 1208.70 | 5440.88 |
| Qwen2.5-3B | DECA | 6912 | 38.39 | 642.836 | 4821.60 |
| Qwen2.5-3B | Dec-LoRA | 6912 | 31.56 | 833.74 | 7333.18 |
| Llama-2-7B | DECA | 6144 | 98.44 | 2105.26 | 2471.95 |
| Llama-2-7B | Dec-LoRA | 6144 | 80.72 | 2809.03 | 4385.06 |
| Llama-3.1-8B | DECA | 6144 | 99.75 | 2,423.08 | 3349.66 |
| Llama-3.1-8B | Dec-LoRA | 6144 | 81.37 | 3176.15 | 4420.80 |
Additional Ablation Studies
We have added results obtained by performing separate ablations of the first- and second-moment corrections on the Qwen2.5-3B model.
| Dataset | Ablation | Accuracy(%) | F1(%) |
|---|---|---|---|
| TFNS | w/o First | 85.36 ± 0.39 | 82.24 ± 0.29 |
| TFNS | w/o Second | 85.92 ± 0.62 | 83.14 ± 0.56 |
| NWGI | w/o First | 50.53 ± 0.74 | 34.58 ± 0.51 |
| NWGI | w/o Second | 51.70 ± 0.51 | 34.57 ± 0.18 |
| MNLI | w/o First | 78.29 ± 1.51 | 75.57 ± 2.31 |
| MNLI | w/o Second | 78.52 ± 1.64 | 75.83 ± 2.67 |
Normalization
In our decentralized learning paradigm, local optimization is driven primarily by the gradient-based Adam moments and . The discrepancy signal serves only as a consensus-guided correction, as described in Sec. 3.2. Under non-IID data, the magnitude of the raw discrepancy may not indicate an ideal correction direction. Directly incorporating an unnormalized discrepancy, particularly when is large, may overwhelm the gradient-based update and distort local optimization. We therefore normalize the discrepancy and use the BMA-specific hyperparameters and to control the correction strength.
Regarding joint normalization, each client has already incorporated information from its neighbors through the parameter aggregation in Eq. (7) before computing . This intermediate variable is then used locally to update the first- and second-order BMA states in Eqs. (9)–(10). Besides, performing "all-together" normalization operations frequently is inconsistent with the decentralized paradigm.
We appreciate the reviewer for these insightful comments. Follows are the first part of our response to the main concerns of the reviewer.
Decentralized Optimization objective
Equation (1) adopts the standard distributed objective function minx∈RdF(x)=N1∑i=1NEζi∼Di[Fi(x;ζi)], which is consistent with the settings in D-PSGD (Lian et al., 2017), QGM (Lin et al., 2021), Momentum Tracking (Takezawa et al., 2023), and Gradient Tracking (Pu and Nedic, 2021). It minimizes the average of the clients' expected losses at a common parameter vector x, where F(x)=N1∑i=1Nfi(x) and fi(x)=Eζi∼Di[Fi(x;ζi)]. The "Final global model" in Algorithm 1 denotes the theoretical output xˉ[T]=N1∑i=1Nxi[T], which also appears in the pseudocodes of D-PSGD (Lian et al., 2017) and GUT (Aketi et al., 2023), where the arithmetic mean of the final local iterates is listed as the output. In practical implementation, DECA maintains only each client's model and its local optimizer states, without requiring an additional global model. Through repeated local optimization and neighbor mixing, information propagates across the network, enabling clients to collaboratively optimize the average objective function while controlling discrepancies among themselves. Theorem 4.5 and Corollary 4.6 demonstrate that, through local updates and partitioned neighbor exchanges, DECA guarantees convergence to a stationarity bound for the objective function given in Equation (1). Consequently, DECA requires no all-reduce operation or final consensus-reaching stage upon training completion, eliminating any additional final model aggregation overhead. The entire training protocol consists solely of the neighbor exchange phase conducted by each client across active blocks, as described in Algorithm 1, with all BMA states maintained locally.
QG-DAdam Implementation
In fact, we have already implemented QG-DAdam and included it in our evaluations. "w/ trivial BMA" in Section 5.3 and Table 3 exactly corresponds to the QG-DAdam implementation, which shares the same block scheduling and Adam optimizer as DECA while matching the training and communication budgets. As described in Section 5.3, the moment states maintained across inner updates by DECA are mi,k[t,r+1]=β1mi,k[t,r+21]+(1−β1)hi,k[t,r] and vi,k[t,r+1]=β2vi,k[t,r+21]+(1−β2)hi,k[t,r]⊙hi,k[t,r], whereas QG-DAdam retains mi,k[t,r+1]=α1mi,k[t,r]+(1−α1)hi,k[t,r] and vi,k[t,r+1]=α2vi,k[t,r]+(1−α2)hi,k[t,r]⊙hi,k[t,r]. Both methods utilize the normalized difference between the local model and the aggregated model when computing the next-round momentum; the key difference lies in DECA leveraging the first- and second-order moments obtained after local updates. As shown in Table 3, compared to the QG-DAdam baseline, DECA improves the average accuracy/F1 score from 58.46/53.83 to 76.32/68.96 on Qwen2.5-3B, and from 78.11/70.61 to 78.91/72.12 on Llama-3.1-8B, clearly demonstrating the advantages of our BMA design.
Granularity
For our main experiments on Llama-3.1-8B, we indeed used a block size of 1 rather than "every two consecutive Transformer layers," meaning each active trainable block contains the parameters of a single Transformer layer. The phrasing "every two consecutive Transformer layers" in Section 5.1 was a typo error. Across all subsequent block-granularity experiments, the granularity consistently represents the number of consecutive Transformer layers per block. We confirm that the results presented in our tables are accurate: the 78.91% average accuracy and 72.12% average F1 score in Table 1 align with the granularity-1 row in Table 7.