acceptodds
Under review as a conference paper at ICLR 2027

DECA: Decentralizing Block-Wise Adam for Efficient LLM Full-Parameter Fine-Tuning on Non-IID Data

Abstract

Fine-tuning large language models (LLMs) in privacy-sensitive and resource-constrained environments remains challenging. Since training data are often distributed across multiple clients, decentralized fine-tuning offers a natural paradigm for collaborative adaptation without a central server. However, enabling full-parameter fine-tuning (FPFT) in this decentralized setting is difficult: FPFT provides strong adaptation capacity but incurs prohibitive resource consumption for billion-scale models. Existing decentralized LLM fine-tuning methods therefore mainly rely on parameter-efficient updates, which improve efficiency but may restrict downstream performance. Moreover, client data are typically non-IID, making decentralized optimization more vulnerable to client drift and unstable convergence. To address these challenges, we propose DECA, a resource-efficient decentralized FPFT framework for LLMs on non-IID data. DECA partitions model parameters into disjoint blocks and performs sequential block-wise Adam optimization, reducing resource consumption while preserving decentralized full-parameter adaptation. To stabilize training, DECA further introduces first- and second-order block-wise moment estimates with fresh local gradient statistics and consensus-derived discrepancy signals. We provide rigorous theoretical analysis and extensive experiments, showing that DECA achieves fast convergence, strong downstream performance, and significant resource efficiency. Our code is available at https://anonymous.4open.science/r/DECA-3F73.

Primary area:
optimization
Keywords:
Decentralized Learning, LLM Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fmDHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

DECA proposes a decentralized large model fine-tuning framework for non IID data, which reduces video memory and computational overhead by updating model parameters block by block, and utilizes consensus information from neighboring models to correct Adam's first-order and second-order momentum estimates, enabling multiple clients to collaboratively fine tune the main parameters of the large language model without a central server.The paper further presents a non-convex convergence analysis and evaluates the method on classification and generation tasks using Qwen and Llama models ranging from 1.5B to 8B parameters.

Critical strengths
  1. Enabling decentralized fine-tuning of large language models without a central parameter server is an important research direction. The combination of limited training resources and non-IID client data presents meaningful challenges beyond conventional centralized fine-tuning.

  2. The proposed framework combines memory-efficient block-wise optimization with peer-to-peer model aggregation. This design provides a plausible approach to updating a large portion of the model parameters while reducing the optimizer-state memory footprint.

  3. The method achieves the best average classification accuracy and F1 score among the evaluated decentralized baselines on both Qwen2.5-3B and Llama-3.1-8B. The stronger non-IID experiment also have good results.

  4. The paper includes classification and generation tasks, multiple communication topologies, different client counts, and block-granularity studies. The measured memory reduction with finer parameter partitioning is consistent with the motivation for block-wise optimization.

Critical weaknesses
  1. The ultimate optimization objective of Formula 1 and Algorithm 1 is the arithmetic mean of all client models. But this global model is not seen in the main methods of this article or in the specific content of Algorithm 1. Although DECA performs neighbor based aggregation during training, each client maintains its own model parameters and optimizer state. In non IID data, the client model does not need to be identical after a limited number of communication rounds. If the final global model is explicitly constructed through additional aggregation processes, the author should explain how this operation can be achieved without central coordination, and whether its communication costs are included in the efficiency analysis

  2. The paper appropriately cites BAdam and Quasi-Global Momentum (QGM) as related work. However, the relationship with QG-DAdam deserves a more direct discussion.Lin et al. (ICML 2021) have explored quasi global momentum in decentralized adaptive optimization and released some variants. DECA differs in its specific two-stage combination of local gradients and consensus discrepancies, as well as its integration with sequential block-wise training.The comparison with Dec-LoRA and DeCAF also changes the trainable parameterization, making it difficult to isolate the optimizer contribution.A direct comparison between DECA and block-wise QG-DAdam, under identical model partitioning, training budgets, and communication conditions, would substantially strengthen the novelty claim.

  3. The resource-efficiency evaluation primarily reports communication volume and computational latency per training step. However, these measurements do not establish the total communication and computation required to complete decentralized fine-tuning. According to Algorithm 1, each of the B parameter blocks undergoes R inner updates and communication operations in each of T global rounds. Thus, the number of block-level communication events per client is TBR. Although communicating one block reduces the payload relative to full-model transmission at an individual step, this reduction must be evaluated together with the number of communication events. Table 8 shows that DECA actually transmits more data per reported communication step than Dec-LoRA and DeCAF. Therefore, the current evidence does not demonstrate lower cumulative communication overhead relative to the main decentralized PEFT baselines. The authors should report the total number of communication events, cumulative transmitted bytes, total training FLOPs, and end-to-end wall-clock time under controlled training budgets.

  4. Section 5.1 states that every two consecutive Transformer layers form a trainable block and reports 32, 36, 28, and 32 blocks for Llama-2-7B, Llama-3.1-8B, Qwen2-1.5B, and Qwen2.5-3B, respectively. But, the Llama-3.1-8B results in Table 1 (78.91% average accuracy and 72.12% average F1) exactly match the granularity-1 results in Table 7, whereas the granularity-2 results are different.This raises uncertainty regarding the actual configuration used in the main experiments.

Questions for authors

1.BMA mechanism: What are the independent contributions of the first- and second-moment corrections? Can the authors provide separate ablations and directly measure client drift or consensus error? 2.For h, the normalization in the text is actually a unit normalization, does it ignore the amplitude difference? Is it better to normalize all h together

2 comments

We appreciate the reviewer for these insightful comments. Follows are the first part of our response to the main concerns of the reviewer.

Decentralized Optimization objective

Equation (1) adopts the standard distributed objective function min⁡x∈RdF(x)=1N∑i=1NEζi∼Di[Fi(x;ζi)]\min_{x \in R^d} F(x) = \frac{1}{N}\sum_{i=1}^{N} E_{\zeta_i \sim D_i}[F_i(x;\zeta_i)], which is consistent with the settings in D-PSGD (Lian et al., 2017), QGM (Lin et al., 2021), Momentum Tracking (Takezawa et al., 2023), and Gradient Tracking (Pu and Nedic, 2021). It minimizes the average of the clients' expected losses at a common parameter vector xx, where F(x)=1N∑i=1Nfi(x)F(x)=\frac{1}{N}\sum_{i=1}^{N}f_i(x) and fi(x)=Eζi∼Di[Fi(x;ζi)]f_i(x)=E_{\zeta_i \sim D_i}[F_i(x;\zeta_i)]. The "Final global model" in Algorithm 1 denotes the theoretical output xˉ[T]=1N∑i=1Nxi[T]\bar{x}^{[T]}=\frac{1}{N}\sum_{i=1}^{N}x_i^{[T]}, which also appears in the pseudocodes of D-PSGD (Lian et al., 2017) and GUT (Aketi et al., 2023), where the arithmetic mean of the final local iterates is listed as the output. In practical implementation, DECA maintains only each client's model and its local optimizer states, without requiring an additional global model. Through repeated local optimization and neighbor mixing, information propagates across the network, enabling clients to collaboratively optimize the average objective function while controlling discrepancies among themselves. Theorem 4.5 and Corollary 4.6 demonstrate that, through local updates and partitioned neighbor exchanges, DECA guarantees convergence to a stationarity bound for the objective function given in Equation (1). Consequently, DECA requires no all-reduce operation or final consensus-reaching stage upon training completion, eliminating any additional final model aggregation overhead. The entire training protocol consists solely of the neighbor exchange phase conducted by each client across active blocks, as described in Algorithm 1, with all BMA states maintained locally.

QG-DAdam Implementation

In fact, we have already implemented QG-DAdam and included it in our evaluations. "w/ trivial BMA" in Section 5.3 and Table 3 exactly corresponds to the QG-DAdam implementation, which shares the same block scheduling and Adam optimizer as DECA while matching the training and communication budgets. As described in Section 5.3, the moment states maintained across inner updates by DECA are mi,k[t,r+1]=β1mi,k[t,r+12]+(1−β1)hi,k[t,r]m_{i,k}^{[t,r+1]} = \beta_1 m_{i,k}^{[t,r+\frac{1}{2}]} + (1-\beta_1)h_{i,k}^{[t,r]} and vi,k[t,r+1]=β2vi,k[t,r+12]+(1−β2)hi,k[t,r]⊙hi,k[t,r]v_{i,k}^{[t,r+1]}=\beta_2 v_{i,k}^{[t, r+\frac{1}{2}]}+(1-\beta_2)h_{i,k}^{[t,r]}\odot h_{i,k}^{[t,r]}, whereas QG-DAdam retains mi,k[t,r+1]=α1mi,k[t,r]+(1−α1)hi,k[t,r]m_{i,k}^{[t,r+1]}=\alpha_1 m_{i,k}^{[t,r]}+(1-\alpha_1) h_{i,k}^{[t,r]} and vi,k[t,r+1]=α2vi,k[t,r]+(1−α2)hi,k[t,r]⊙hi,k[t,r]v_{i,k}^{[t,r+1]}=\alpha_2 v_{i,k}^{[t,r]}+(1-\alpha_2)h_{i,k}^{[t,r]}\odot h_{i,k}^{[t, r]}. Both methods utilize the normalized difference between the local model and the aggregated model when computing the next-round momentum; the key difference lies in DECA leveraging the first- and second-order moments obtained after local updates. As shown in Table 3, compared to the QG-DAdam baseline, DECA improves the average accuracy/F1 score from 58.46/53.83 to 76.32/68.96 on Qwen2.5-3B, and from 78.11/70.61 to 78.91/72.12 on Llama-3.1-8B, clearly demonstrating the advantages of our BMA design.

Granularity

For our main experiments on Llama-3.1-8B, we indeed used a block size of 1 rather than "every two consecutive Transformer layers," meaning each active trainable block contains the parameters of a single Transformer layer. The phrasing "every two consecutive Transformer layers" in Section 5.1 was a typo error. Across all subsequent block-granularity experiments, the granularity consistently represents the number of consecutive Transformer layers per block. We confirm that the results presented in our tables are accurate: the 78.91% average accuracy and 72.12% average F1 score in Table 1 align with the granularity-1 row in Table 7.

We appreciate the reviewer for these insightful comments. Follows are the second part of our response to the main concerns of the reviewer.

Total Communication and Computation

In our experiments, we ensure that all baseline methods have the same number of updates and communication rounds, resulting in an identical total number of communication events. The table below reports the total number of communication events, cumulative bytes transmitted, total training FLOPs, and actual end-to-end wall-clock time for both Dec-LoRA and DECA across four model-task configurations. (Specifically, the sequence length is set to 1024 for Qwen2-1.5B and Qwen2.5-3B, and 512 for Llama-2-7B and Llama-3.1-8B; all other training-related parameters remain consistent.) As shown, although DECA incurs a higher cumulative communication overhead than Dec-LoRA, it outperforms Dec-LoRA in terms of both total training FLOPs and actual end-to-end wall-clock time.

ModelMethodsTotal communication eventsCumulative transmitted (TiB)Total training (PFLOPs)End-to-end wall-clock time (s)
Qwen2-1.5BDECA537618.09938.8063605.6
Qwen2-1.5BDec-LoRA537615.0741208.705440.88
Qwen2.5-3BDECA691238.39642.8364821.60
Qwen2.5-3BDec-LoRA691231.56833.747333.18
Llama-2-7BDECA614498.442105.262471.95
Llama-2-7BDec-LoRA614480.722809.034385.06
Llama-3.1-8BDECA614499.752,423.083349.66
Llama-3.1-8BDec-LoRA614481.373176.154420.80

Additional Ablation Studies

We have added results obtained by performing separate ablations of the first- and second-moment corrections on the Qwen2.5-3B model.

DatasetAblationAccuracy(%)F1(%)
TFNSw/o First85.36 ± 0.3982.24 ± 0.29
TFNSw/o Second85.92 ± 0.6283.14 ± 0.56
NWGIw/o First50.53 ± 0.7434.58 ± 0.51
NWGIw/o Second51.70 ± 0.5134.57 ± 0.18
MNLIw/o First78.29 ± 1.5175.57 ± 2.31
MNLIw/o Second78.52 ± 1.6475.83 ± 2.67

Normalization

In our decentralized learning paradigm, local optimization is driven primarily by the gradient-based Adam moments mi,k[t,r+12]m_{i,k}^{[t,r+\frac{1}{2}]} and vi,k[t,r+12]v_{i,k}^{[t,r+\frac{1}{2}]}. The discrepancy signal hi,k[t,r]h_{i,k}^{[t,r]} serves only as a consensus-guided correction, as described in Sec. 3.2. Under non-IID data, the magnitude of the raw discrepancy Δi,k[t,r]=xi,k[t,r]−xi,k[t,r+1]\Delta_{i,k}^{[t,r]} = x_{i,k}^{[t,r]}-x_{i,k}^{[t,r+1]} may not indicate an ideal correction direction. Directly incorporating an unnormalized discrepancy, particularly when ∥Δi,k[t,r]∥2\Vert \Delta_{i,k}^{[t,r]}\Vert_2 is large, may overwhelm the gradient-based update and distort local optimization. We therefore normalize the discrepancy and use the BMA-specific hyperparameters β1\beta_1 and β2\beta_2 to control the correction strength.

Regarding joint normalization, each client has already incorporated information from its neighbors through the parameter aggregation in Eq. (7) before computing hi,k[t,r]h_{i,k}^{[t,r]}. This intermediate variable is then used locally to update the first- and second-order BMA states in Eqs. (9)–(10). Besides, performing "all-together" normalization operations frequently is inconsistent with the decentralized paradigm.

Reviewer_bCCbOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces DECA, a decentralized framework for full-parameter LLM fine-tuning on heterogeneous (non-IID) client data without a central server. It updates and communicates one parameter block at a time, reducing memory and communication costs while allowing the entire model to be fine-tuned over time. To reduce client drift, DECA adjusts Adam’s moment estimates using local gradients and information from neighboring clients. The paper presents experiments on 1.5B–8B models showing stronger overall classification and generation performance than decentralized baselines, with classification results approaching centralized block-wise fine-tuning.

Critical strengths
  1. The paper is easy to follow and contains useful results across different tasks and settings.

  2. The paper clearly compares with the most important baselines in decentralized LLM finetuning.

  3. The paper provides sensitivity results for the number of clients and the network topology.

Critical weaknesses
  1. My main concern is that I do not think the paper's methodology is particularly specific to LLMs. The Block-wise updates, adaptive moment estimation, and consensus-based correction are applicable to general decentralized problems. It would be great if the others could clearly explain which parts of the paper's algorithmic contribution are particularly relevant to LLM fine-tuning. For example, baselines such as DECAF and Dec-LoRA focus on PEFT methods like LoRA, which makes their methods favorable for LLMs.

  2. The paper should clearly distinguish DECA from the closest existing, more classical optimization methods, particularly decentralized BAdam and block-wise implementation of QG-DAdam. Comparing the proposed method with LoRA-based methods cannot separate the benefit of the optimizer choices.

  3. The evaluation is mainly based on three classification datasets, which can limit the benefit of the proposed algorithm for broader LLM application. Additional generation tasks can address this concern.

  4. The ablations do not isolate the effect of the first and second moment correction. Ablation studies are needed to establish whether both components are necessary. Also, sensitivity to the number of updates per block is unexplored.

Questions for authors
  1. Can the authors report the number of trainable parameters for the proposed methods and baselines? I think the paper proposed full fine-tuning of LLMs, which is not fair to compare with LoRA-based methods.

  2. Can the authors explain how the algorithm is particularly beneficial for LLMs rather than a more general problem?

1 comment

We appreciate the reviewer for these insightful comments. Follows are our response to the main concerns of the reviewer.

LLMs Sepcification

DECA is particularly beneficial for LLM fine-tuning because its design specifically targets the parameter scale, and modular architecture. Similar in motivation to BAdam (Luo et al., 2024), updating the entire model simultaneously with Adam at the billion-parameter scale requires maintaining gradients and two moment buffers for all trainable parameters, which creates a substantial memory bottleneck on resource-constrained devices. Leveraging the natural stacked structure of Transformers in LLMs, DECA treats individual or multiple consecutive layers as natural blocks with clear parameter boundaries. Besides, DECA maintains first- and second-order moment estimates solely for the current block and reinitializes these moment states at the beginning of each block access (see Section 3.2). Consequently, assuming the modules are of same size and consistent state precision, the storage space required for the moment states can be reduced to approximately 1/B1/B of the storage space that would be needed for the full model.

QG-DAdam Implementation

In fact, we have already implemented QG-DAdam and included it in our evaluations. "w/ trivial BMA" in Section 5.3 and Table 3 exactly corresponds to the QG-DAdam implementation, which shares the same block scheduling and Adam optimizer as DECA while matching the training and communication budgets. As described in Section 5.3, the moment states maintained across inner updates by DECA are mi,k[t,r+1]=β1mi,k[t,r+12]+(1−β1)hi,k[t,r]m_{i,k}^{[t,r+1]} = \beta_1 m_{i,k}^{[t,r+\frac{1}{2}]} + (1-\beta_1)h_{i,k}^{[t,r]} and vi,k[t,r+1]=β2vi,k[t,r+12]+(1−β2)hi,k[t,r]⊙hi,k[t,r]v_{i,k}^{[t,r+1]}=\beta_2 v_{i,k}^{[t, r+\frac{1}{2}]}+(1-\beta_2)h_{i,k}^{[t,r]}\odot h_{i,k}^{[t,r]}, whereas QG-DAdam retains mi,k[t,r+1]=α1mi,k[t,r]+(1−α1)hi,k[t,r]m_{i,k}^{[t,r+1]}=\alpha_1m_{i,k}^{[t,r]}+(1-\alpha_1) h_{i,k}^{[t,r]} and vi,k[t,r+1]=α2vi,k[t,r]+(1−α2)hi,k[t,r]⊙hi,k[t,r]v_{i,k}^{[t,r+1]}=\alpha_2v_{i,k}^{[t,r]}+(1-\alpha_2)h_{i,k}^{[t,r]}\odot h_{i,k}^{[t, r]}. Both methods utilize the normalized difference between the local model and the aggregated model when computing the next-round momentum; the key difference lies in DECA leveraging the first- and second-order moments obtained after local updates.

Additional Generation tasks

To better demonstrate the effectiveness of our method on generation tasks, we present experiments on GSM8K, which is an open-source dataset consisting of 8.5K linguistically diverse grade-school mathematical word problems, designed for natural language reasoning and answer generation. The results are presented below. As shown, our DECA algorithm consistently achieves the best performance compared to both the baseline and ablation variants.

MethodACC (%)
DAdapter41.42 ± 0.41
Dec-LoRA53.62 ± 0.18
DeCAF53.65 ± 0.41
w/ trivial BMA14.02 ± 1.69
w/o BMA54.47 ± 0.31
DECA61.64 ± 0.20

Additional Ablation Studies

We present results obtained by performing separate ablations of the first- and second-moment corrections and different updates per block on the Qwen2.5-3B model. The results are presented below.

DatasetAblationAccuracy (%)F1 (%)
TFNSw/o First85.36 ± 0.3982.24 ± 0.29
TFNSw/o Second85.92 ± 0.6283.14 ± 0.56
NWGIw/o First50.53 ± 0.7434.58 ± 0.51
NWGIw/o Second51.70 ± 0.5134.57 ± 0.18
MNLIw/o First78.29 ± 1.5175.57 ± 2.31
MNLIw/o Second78.52 ± 1.6475.83 ± 2.67
DatasetRoundsAccuracy (%)F1 (%)
TFNS3688.16 ± 0.2485.33 ± 0.34
TFNS6089.17 ± 0.2786.52 ± 0.33
NWGI3654.77 ± 0.1235.20 ± 0.14
NWGI6056.32 ± 0.2239.14 ± 0.14
MNLI3682.36 ± 0.7481.36 ± 0.83
MNLI6088.67 ± 0.1788.49 ± 0.17

Trainable parameters

We now report the number of trainable parameters for the DECA and Dec-LoRA methods across four models.

ModelDECADec-LoRA
Qwen2-1.5B33,030,14427,525,120
Qwen2.5-3B54,525,95244,826,624
Llama-2-7B157,286,400128,974,848
Llama-3.1-8B159,383,552130,023,424

Taking the Qwen2.5-3B model as an example, DECA has 54.5M trainable parameters per transformer block, whereas Dec-LoRA has 44.8M trainable parameters on LoRA. Since DECA does indeed have more trainable parameters than Dec-LoRA, in order to isolate the effect of the number of trainable parameters, we further evaluate Dec-LoRA on Qwen2.5-3B model with ranks r=64r=64, in addition to the default setting r=32r=32 reported in our original submission. As shown in the results below, increasing the rank generally improves model accuracy. Nevertheless, despite using substantially more trainable parameters (89.6M) with r=64r=64 , Dec-LoRA still underperforms DECA on Average Accuracy and F1 score, based on the corresponding results reported in Table 1 of our submission.

DatasetAccuracy (%)F1 (%)
NWGI53.29 ± 1.7035.85 ± 0.30
TFNS89.16 ± 0.6286.39 ± 0.57
MNLI82.66 ± 1.4682.04 ± 1.18
Avg75.0468.09
Reviewer_nHkrOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DECA addresses decentralized fine-tuning of large language models on non-IID client data. Each client updates original model parameters one block at a time and exchanges the active block with its neighbors. Block-wise moment approximations (BMAs) combine fresh local gradient statistics with normalized model movement after neighbor averaging to modify Adam’s first- and second-moment estimates. The implementation trains the Transformer core while freezing embeddings and the language-model head. The paper presents a convergence analysis and evaluates four models spanning 1.5B–8B parameters on classification and generation tasks, comparing DECA with decentralized parameter-efficient methods and centralized BAdam.

Critical strengths
  1. Useful approach with competitive classification results. Maintaining gradients and optimizer states only for the active block reduces their memory requirements compared with full-model Adam and enables adaptation of original Transformer parameters without an aggregation server. Table 1 reports the highest average accuracy and F1 among the evaluated decentralized methods on both models. On Llama-3.1-8B, DECA achieves 78.91% average accuracy, compared with 77.33% for Dec-LoRA and 80.13% for centralized BAdam. These results support the relevance of the approach.

  2. Ablations support the inclusion of BMA. Removing BMA reduces average classification accuracy from 76.32% to 69.56% on Qwen2.5-3B and from 78.91% to 72.90% on Llama-3.1-8B (Table 3). Table 6 provides further evidence under stronger heterogeneity. The advantage over the trivial BMA variant is model-dependent: 17.86 percentage points on Qwen, but only 0.80 points on Llama. The results support BMA’s usefulness, although uncertainty estimates would strengthen the comparison between its variants.

Critical weaknesses
  1. Training-budget alignment and experimental configuration are unclear. Algorithm 1 implies T × B × R local optimizer updates and neighbor-exchange phases per client, but the paper does not clearly explain how training budgets were aligned across methods. Total processed examples or tokens and method-specific tuning procedures are insufficiently documented. Section 5.1 also specifies two Transformer layers per block, while Table 7’s granularity-one row reproduces the Llama-3.1-8B results in Table 1. Since block granularity affects training exposure and resource consumption, these settings need reconciliation before the comparisons can be assessed confidently.

  2. Efficiency claims require consistent accounting and clearer timing measurements. Appendix B.1 gives a per-round communication cost of O(d × R × Nmax / B), whereas Algorithm 1 visits all B blocks, implying O(d × R × Nmax) per complete sweep under the equal-block approximation. Table 8 reports 2,100 MB per step for DECA versus 1,722 MB for Dec-LoRA on Llama-2-7B. It also reports forward latencies of 140.84 ms versus 323.60 ms. Since both methods require a forward pass through the full model, this substantial timing difference warrants explanation. Comparable measurement settings, total training time, and transmitted bytes at comparable performance are needed to establish the overall efficiency gain.

Questions for authors
  1. Training configuration: What batch sizes, sequence lengths, LoRA ranks, total optimizer steps, and processed tokens were used for each method? How were hyperparameters selected, and which block granularity produced the headline results?
  2. Efficiency measurements: How should Appendix B.1’s communication cost be reconciled with Algorithm 1? What explains the forward-latency difference in Table 8, and were batch size, sequence length, precision, and profiling procedures matched? Can complete training costs be reported at comparable test performance?
  3. Evaluation uncertainty: Can the authors report variability and the number of runs for the BMA ablations and generation results, and identify the exact judge-model version used for generation evaluation?
1 comment

We appreciate the reviewer for these insightful comments. Follows are our response to the main concerns of the reviewer.

Reconciliation of Experiments

In Algorithm 1, each client performs T×B×RT \times B \times R local optimizer updates and neighbor exchange phases; we strictly matched this budget in all decentralized comparison methods by ensuring that, under the identical batch size 4 and sequence length 512, they also undergo T×B×RT \times B \times R local optimizer updates and neighbor exchanges. Furthermore, for all LoRA-based baselines, the adapter rank hyperparameters were set to rank=32rank = 32 and α=64\alpha = 64 for Qwen2-1.5B and Qwen2.5-3B, and rank=64rank = 64 and α=128\alpha = 128 for Llama-2-7B and Llama-3.1-8B, with a uniform dropout rate of 0.050.05. All baseline methods adopted the default Adam parameters consistent with DECA (as described in Section 5.1). Regarding the granularity mismatch, our main experiments indeed employed a granularity of 1, meaning each active trainable block contains the parameters of a single Transformer layer, and the phrasing "every two consecutive Transformer layers" in Section 5.1 was a typo error; all subsequent granularity experiments follow this convention, where the granularity represents the number of consecutive Transformer layers per block.

Efficiency

We re-evaluated the performance of all baseline methods under comparable settings, using actual CUDA events execution times for both the forward and backward passes. Data for the Qwen2-1.5B and Qwen2.5-3B models were collected on an NVIDIA 5090 GPU, while data for the Llama-2-7B and Llama-3.1-8B models were collected on an NVIDIA A800 GPU. All experiments were conducted with a batch size of 44 and a sequence length of 512512. The results are summarized below. We will incorporate this result into subsequent revisions.

MethodsModelTotal Forward (s)Total Backward (s)Total Exchange (s)Total StepsAvg Forward perstep (ms)Avg Backward perstep (ms)Avg Agg & Up perstep (ms)
DECALlama-3.1-8B1053.83705.8488.376144171.52114.8814.38
Dec-LoRALlama-3.1-8B1404.021674.25149.046144228.52272.5024.26
DecafLlama-3.1-8B1418.271671.4517353.646144230.84272.052824.49
DadapterLlama-3.1-8B1118.621354.75128.206144182.07220.5020.87
DECALlama2-7B989.16638.8887.866144160.99103.9814.30
Dec-LoRALlama2-7B1325.131565.54150.206144215.68254.8124.45
DecafLlama2-7B1326.171564.2717871.316144215.85254.602908.74
DadapterLlama2-7B1046.731278.49128.446144170.37208.0920.91
DECAQwen2.5-3B375.84272.60149.82691254.3739.4421.67
Dec-LoRAQwen2.5-3B537.28626.58327.46691277.7390.6547.38
DecafQwen2.5-3B527.31632.9110685.26691276.2991.571545.90
DadapterQwen2.5-3B409.49517.45194.83691259.2474.8628.19
DECAQwen2-1.5B255.92228.2543.49537647.6042.468.09
Dec-LoRAQwen2-1.5B341.22499.1576.05537663.4792.8514.15
DecafQwen2-1.5B345.29500.094770.02537664.2393.02887.28
DadapterQwen2-1.5B262.65391.6871.11537648.8672.8613.23

Furthermore, we also calculated the FLOPs for the forward and backward passes of the DECA and Dec-LoRA methods across all four models. For the backward pass, the reported FLOPs represent the average value calculated over a complete cycle of block activations in DECA. We use TFLOPS as the unit of measurement.

ModelBlocksDECA avg forwardDECA avg backwardDec-LoRA avg forwardDec-LoRA avg backward
Qwen2-1.5B289.756.079.9210.34
Qwen2.5-3B3619.4211.4519.7020.40
Llama-2-7B3227.6115.2228.1429.01
Llama-3.1-8B3231.2918.0131.8232.80

Additional Results

We update results of ablation studies and Alpaca tasks based on 5 runs. The evaluation model we actually used for Alpaca measurements is GPT-4o.

Qwen2.5-3B

DatasetMethodAccuracy (%)F1 (%)
MNLIw/o BMA77.90±1.7874.72±2.76
TFNSw/o BMA82.54±0.7979.86±0.78
NWGIw/o BMA49.30±0.5332.77±1.14
MNLIw. trivial BMA69.27±1.5966.15±4.05
TFNSw. trivial BMA78.36±4.8676.23±4.39
NWGIw. trivial BMA30.07±4.4819.98±4.09

Llama-3.1-8B

DatasetMethodAccuracy (%)F1 (%)
MNLIw/o BMA77.85±0.7375.10±1.16
TFNSw/o BMA84.89±1.6782.83±1.44
NWGIw/o BMA54.16±0.4437.97±0.45
MNLIw. trivial BMA87.23±0.3386.94±0.39
TFNSw. trivial BMA89.19±0.6986.14±0.87
NWGIw. trivial BMA56.21±0.4736.14±1.07

Qwen2-1.5B

MethodVICMT-1MT-2MT
Dec-LoRA5.49±0.024.45±0.073.04±0.103.75±0.08
DeCAF5.65±0.044.83±0.033.23±0.074.03±0.04
DECA5.67±0.034.88±0.023.33±0.024.10±0.01
Dec-Adapter5.30±0.044.26±0.052.74±0.073.50±0.05

Llama-2-7B

MethodVICMT-1MT-2MT
Dec-LoRA5.54±0.054.32±0.082.84±0.073.58±0.07
DeCAF5.90±0.044.45±0.092.97±0.053.71±0.05
DECA5.91±0.044.59±0.082.94±0.033.77±0.05
Dec-Adapter4.72±0.113.78±0.122.52±0.103.15±0.11

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.