acceptodds
Under review as a conference paper at ICLR 2027

Breaking the Memory Barrier for On-Device LLM Adaptation via Dynamic Layer Pruning

Abstract

Federated fine-tuning holds the promise of privacy-preserving Large Language Model (LLM) adaptation, yet the prohibitive memory costs limit participation from resource-constrained devices. In this paper, we propose FedPruner, a novel paradigm that enables efficient training via intelligent, on-the-fly layer pruning. FedPruner flexibly prunes the global model, creating personalized submodels based on device memory constraints. It employs a macro-micro synergistic pruning framework: a macro-level Functionality-Driven Layer Orchestration mechanism groups layers, while a micro-level Importance-Aware Layer Selection strategy prunes within groups to build device-specific submodels. We further introduce a fine-grained variant that independently prunes Multi-Head Attention and Feed-Forward Network components to precisely preserve critical architectural elements. Extensive experiments demonstrate that FedPruner significantly outperforms state-of-the-art methods with average accuracy gains of up to 11.11%. Moreover, it maintains strong robustness under varying memory constraints, yielding a 1.98% average performance improvement while reducing peak memory usage by 75%.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Federated Fine-Tuning, Layer Pruning, Memory Efficiency

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zmhQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies federated LLM fine-tuning under heterogeneous memory constraints and proposes dynamic layer pruning to construct submodels tailored to client resources. Its macro-level module, FDLO, constructs a graph using CKA similarities between layer outputs on local data and partitions the layers into as many groups as the client can retain layers. Its micro-level module, IALS, derives within-group sampling probabilities from differences between layer inputs and outputs, selecting one representative layer per group. The resulting submodels undergo local LoRA fine-tuning, after which updates are uploaded for aggregation. The fine-grained variant, FedPruner+, extends pruning to individual MHA and FFN components. Through instruction tuning on Alpaca-GPT4, closed-ended and open-ended evaluations, memory-budget experiments, module ablations, and resource measurements, the paper reports improvements in model performance, client participation, and training efficiency.

Critical strengths

S1. The problem is well defined, and the method directly addresses the memory burden of base-model weights. The paper clearly distinguishes reducing the number of trainable parameters from reducing the memory required to keep the model resident. Its profiling demonstrates that, under the tested configurations, optimizing LoRA modules alone does not eliminate the memory overhead of the base-model weights. FedPruner instead adapts the model structure used in each client’s forward and backward computations, allowing resource-constrained clients to contribute their local data. The proposed structured pruning approach directly addresses the identified bottleneck and has clear practical relevance to federated fine-tuning under heterogeneous resource constraints.

S2. Comparisons with FedRA show that the gains cannot be attributed solely to enabling more clients to participate. In Table 1, both FedRA and FedPruner allow all devices to participate, yet FedPruner improves average closed-ended benchmark performance by 1.71 and 4.30 percentage points on LLaMA2-7B and LLaMA2-13B, respectively. Table 2 further compares the methods under matched budgets expressed as fractions of the global model. For example, under the 25% budget for the 13B model, FedPruner achieves 50.56 compared with FedRA’s 44.99. These comparisons help isolate the value of the submodel construction strategy and provide key evidence for the methodological contribution.

S3. Ablation studies directly support the contributions of macro-level grouping and micro-level selection. Table 3 removes FDLO, IALS, and both modules. On LLaMA2-13B, average performance decreases by 2.85, 1.29, and 4.30 percentage points, respectively, with consistent trends on the 7B model. These results support contributions from both modules rather than gains driven by only one component. In addition, Table 4 reports that FedPruner reduces training time from 2.10 to 1.92 hours and communication volume from 3.65 to 3.47 GB relative to FedRA, indicating that its accuracy improvements do not come at the expense of higher time or communication costs in this experiment.

Critical weaknesses

W1. Direct evidence for the explanation of the pruning mechanism remains limited. The method groups layers according to output similarity and selects a representative layer from each group to preserve hierarchical information-processing capabilities. Figure 6 in Section 4.4 shows a selection pattern favoring early and late layers, while the ablations in Table 3 support the contributions of both modules to final performance. These findings support the effectiveness of the selection strategy, but do not directly explain why grouping by representational similarity preserves essential hierarchical processing capabilities under aggressive pruning.

W2. The additional benefits of client-specific data adaptation have not been clearly isolated. Section 3.1 presents local similarity computation as a mechanism for handling data heterogeneity, allowing clients to construct submodels according to their own data distributions. The overall results and memory-budget experiments in the main paper support the framework’s effectiveness under resource constraints. However, they do not clearly distinguish gains from generally better layer selection from those attributable to adaptation to individual clients’ data distributions.

Questions for authors

Q1. Under aggressive pruning, to what extent does FedPruner’s advantage come from constructing submodels that better preserve the original model’s capabilities, versus subsequent dynamic training compensating for pruning-induced degradation? Could the authors draw on existing submodel performance results to further explain why grouping layers by similarity and retaining one representative per group preserves important inter-layer interactions?

Q2. How does FedPruner’s advantage over FedRA change as the data distributions across clients become more heterogeneous?

1 comment
AuthorsOpenReview ↗

Response to Reviewer zmhQ

We sincerely thank the reviewer for the constructive and positive feedback. Below are our point-by-point responses to your concerns.

W1 & Q1: Why grouping and representative selection help under aggressive pruning.

Thank you for this insightful question. Our rationale is that individual-layer redundancy does not imply that multiple layers can be safely removed together. Several layers may provide overlapping representations, making each appear individually dispensable; removing them jointly can eliminate an intermediate processing stage needed by subsequent layers. Figures 2 and 3 motivate this concern: limited degradation from individual-layer removal does not translate into reliable performance under multi-layer pruning.

FDLO addresses this joint-selection problem by grouping layers according to output similarity on local data and retaining one representative from each group. This reduces redundancy within similar regions while preventing any identified representation group from being omitted entirely. With representatives retained in their original order, the submodel maintains a compressed processing path across different representation stages, helping preserve the composition of earlier-stage representations and later-stage transformations. IALS then favors higher-scoring representatives within each group.

Appendix G illustrates this allocation. In Table 6, the 32-layer model retains layers 1–5 and 27–32 as singleton representatives, together with one representative from the highly similar middle region, layers 6–26. The ablations in Table A further support the complementary roles of the two components. Removing FDLO decreases average accuracy by 1.22 and 2.85 on LLaMA2-7B and LLaMA2-13B, respectively. The corresponding decreases are 0.57 and 1.29 without IALS, and 1.71 and 4.30 without both components. The largest decline occurs when both are removed, providing further evidence for the effectiveness of our coordinated submodel construction strategy.

Table A: Ablation analysis of FedPruner.

ModelMethodTruthfulQAMMLUIFEvalBBHAvg.
LLaMA2-7BFedPruner49.9543.6433.7240.6541.99
w/o FDLO48.1442.9332.4439.5840.77 (−1.22)
w/o IALS49.3843.0232.8740.4241.42 (−0.57)
w/o FDLO & IALS47.6642.4131.6739.3640.28 (−1.71)
LLaMA2-13BFedPruner56.1358.0945.6248.4652.08
w/o FDLO53.2455.7641.4746.4549.23 (−2.85)
w/o IALS54.8256.9443.6447.7450.79 (−1.29)
w/o FDLO & IALS51.6954.9339.1145.3947.78 (−4.30)

W2 & Q2: Advantage over FedRA as client data become more heterogeneous.

Thank you for this constructive comment. To directly examine how the advantage over FedRA changes with data heterogeneity, we conducted additional experiments on YELP-P and AGNEWS, using RoBERTa as the global model. These benchmarks provide explicit class labels, allowing us to construct three client-data distribution settings: IID, Non-IID, and Severe Non-IID.

As shown in Table B, FedPruner consistently outperforms FedRA, and its advantage widens as heterogeneity increases. On YELP-P, the accuracy gap grows from 8.33 under IID to 10.48 under Non-IID and 14.11 under Severe Non-IID. On AGNEWS, the corresponding gains increase from 8.55 to 11.89 and 16.13. These results provide empirical evidence that FedPruner better accommodates increasing client-data heterogeneity. We will include these additional results in the revised manuscript.

Table B: Accuracy under different client-data distributions.

BenchmarkMethodIIDNon-IIDSevere Non-IID
YELP-PFedRA79.2174.3565.57
FedPruner87.5484.8379.68
AGNEWSFedRA83.4977.5269.12
FedPruner92.0489.4185.25

Finally, thanks again for your time and helpful comments. We hope your concerns are well addressed.

Reviewer_2ECZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies memory-efficient federated fine-tuning of large language models on resource-constrained devices. The authors propose FedPruner, which dynamically constructs device-specific submodels through a macro-micro layer pruning framework. Specifically, it groups layers based on their functional similarity and then selects representative layers according to their importance. The authors also introduce FedPruner+, a fine-grained variant that independently prunes attention and feed-forward components. Experiments across different LLMs, benchmarks, and memory constraints show that the proposed methods generally outperform the evaluated baselines in model performance and resource efficiency.

Critical strengths
  1. The paper addresses an important practical challenge in federated LLM fine-tuning: the high memory requirements of large models prevent many resource-constrained devices from participating in training.

  2. The proposed macro-micro pruning framework provides a structured way to construct submodels under different device memory constraints.

Critical weaknesses
  1. Unclear compatibility of updates from differently pruned client models. FedPruner constructs personalized submodels based on each client's local data and memory constraints, so different clients may retain different layers. Although the authors introduce a layer-wise LoRA aggregation mechanism, the same layer may receive different input representations and gradients under different pruned architectures. Therefore, even when two clients update the same layer of the original model, their updates may not follow consistent optimization directions for the full global model. The paper does not sufficiently explain how these differences affect global training, especially under heterogeneous client data distributions.

  2. Limited justification for the importance-aware layer selection strategy. The authors measure layer importance using feature similarity between consecutive layers and convert the scores into selection probabilities through softmax. However, a larger change in feature representations does not necessarily mean that a layer contributes more to the final task performance.

  3. Unclear control of LoRA rank growth during aggregation. This paper aggregate LoRA parameters from different clients through concatenation rather than directly averaging the parameter matrices. However, concatenating low-rank matrices can increase the rank of the aggregated LoRA module. For example, combining updates of rank (r) from two clients may result in a representation with rank (2r). The paper does not clearly explain how this rank growth is controlled across communication rounds, or whether any rank compression, truncation, or reinitialization is applied.

  4. Insufficient clarification of the memory profiling results. Figure 1(c) reports that model parameters account for over 90% of the training memory, while intermediate activations occupy only a small fraction. The authors use this observation to motivate reducing model parameters rather than optimizing activation memory. However, such a highly skewed memory breakdown depends strongly on the training configuration and may not be representative of common LoRA fine-tuning settings. In particular, the paper does not clearly explain which memory optimization techniques were used in this analysis. While reducing parameter memory is a reasonable motivation, presenting these results without sufficient context may give readers a misleading impression of the general memory bottlenecks in LLM fine-tuning.

Questions for authors
  1. Could the authors further explain how LoRA updates from different pruned submodels can be effectively aggregated into the full global model? It would be helpful to analyze the differences in layer selection across clients and evaluate the stability and effectiveness of the aggregation strategy under varying degrees of data heterogeneity.

  2. Could the authors provide more evidence that the proposed importance score reflects a layer's actual contribution to model performance? In particular, it would be helpful to compare the current probabilistic strategy with uniform random selection within each group and deterministic selection of the highest-importance layer, and to examine the effect of different softmax temperatures.

  3. Could the authors clarify how LoRA concatenation in Equation (5) maintains a fixed or controlled rank across multiple communication rounds? If rank compression, truncation, or reinitialization is used, could the authors explain its implementation and its impact on model performance, memory usage, and communication cost?

  4. Could the authors clarify the model precision, quantization settings, and activation memory optimization techniques used in Figure 1? In particular, does the memory breakdown show a similar trend when using INT4 quantization, as in the later experiments, together with common memory optimization techniques?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 2ECZ (1/2)

We sincerely thank the reviewer for the constructive feedback. Below are our point-by-point responses to your concerns.

W1 & Q1: Compatibility of updates from differently pruned architectures.

Thank you for highlighting this issue. We agree that even updates to the same original layer can differ across clients, because different surrounding architectures change its input representations and backpropagation path. This can introduce architecture-induced optimization bias.

FDLO is designed to mitigate this discrepancy by preserving coverage of the model's hierarchical processing stages within each submodel. Selecting representatives from different similarity groups reduces the risk of omitting entire representation stages and helps retain a sequence of complementary transformations. The rationale is to keep shared layers embedded in a coherent, compressed processing hierarchy, thereby encouraging more compatible optimization directions across submodels. This is a design objective rather than a guarantee of gradient alignment. We will clarify this rationale and its limitations in the revised manuscript.

To assess model performance under increasing data heterogeneity, we conducted additional experiments using RoBERTa on YELP-P and AGNEWS under IID, Non-IID, and Severe Non-IID distributions. As Table A shows, FedPruner consistently outperforms FedRA and exhibits a smaller accuracy decline as heterogeneity increases. Its advantage grows from 8.33 to 14.11 on YELP-P and from 8.55 to 16.13 on AGNEWS, from IID to Severe Non-IID. These results support the framework's empirical effectiveness despite architectural differences across clients.

Table A: Accuracy under different client-data distributions.

BenchmarkMethodIIDNon-IIDSevere Non-IID
YELP-PFedRA79.2174.3565.57
FedPruner87.5484.8379.68
AGNEWSFedRA83.4977.5269.12
FedPruner92.0489.4185.25

W2 & Q2: Effect of the importance-aware layer selection strategy.

Thank you for this insightful comment. We agree that a larger representational change does not necessarily imply a greater contribution to final task performance. Our rationale is to use input–output changes measured on local data as a task-conditioned proxy for representational impact. Within each FDLO group, this provides a preference for layers that more strongly transform the observed representations. Probabilistic selection then favors these layers while allowing alternative representatives to be updated and different submodel combinations to be explored.

To evaluate this design, we compared IALS with uniform within-group sampling and deterministic top-1 selection. As shown in Table B, replacing IALS with uniform sampling decreases average accuracy by 0.57 and 1.29 on LLaMA2-7B and LLaMA2-13B, respectively. Deterministic top-1 selection produces larger decreases of 0.73 and 1.63. These comparisons support both the importance-based preference over uniform sampling and the benefit of retaining stochastic exploration beyond greedy selection.

The softmax temperature controls this balance:

pn(τ)=exp⁡(σn/τ)∑m∈Gkexp⁡(σm/τ).p_n(\tau)=\frac{\exp(\sigma_n/\tau)}{\sum_{m\in G_k}\exp(\sigma_m/\tau)}.

Actually, larger temperatures flatten the distribution toward uniform sampling, while smaller temperatures concentrate probability on the highest-scoring layers. We will clarify the rationale and heuristic nature of the score, explicitly state the temperature setting, and include these additional comparisons in the revised manuscript.

Table B: Analysis of the importance-aware layer selection strategy.

ModelMethodTruthfulQAMMLUIFEvalBBHAvg.
LLaMA2-7BFedPruner49.9543.6433.7240.6541.99
w/o IALS (Uniform)49.3843.0232.8740.4241.42 (−0.57)
Deterministic top-149.2142.9632.6940.1841.26 (−0.73)
LLaMA2-13BFedPruner56.1358.0945.6248.4652.08
w/o IALS (Uniform)54.8256.9443.6447.7450.79 (−1.29)
Deterministic top-154.5356.6843.2547.3350.45 (−1.63)
AuthorsOpenReview ↗

Response to Reviewer 2ECZ (2/2)

W3 & Q3: LoRA rank after concatenation, across clients and communication rounds.

Thank you for raising this question. The key distinction is between the stacked width used for aggregation and the local adapter width used for training. Following FLoRA[1], we merge each round's aggregated update into the backbone and reinitialize local adapters before the next round, preventing cumulative growth in trainable adapter width.

Within-round aggregation. Let Snt\mathcal{S}_n^t denote the clients retaining and updating layer nn in round tt. Each client trains a fresh adapter of width rsr_s while keeping the backbone frozen. With LoRA scaling absorbed into the factors, Equation (5) exactly represents ΔWnt=Bcat,ntAcat,nt=∑s∈Sntλs,ntBs,ntAs,nt,λs,nt=∣Ds∣∑j∈Snt∣Dj∣.\Delta W_n^t = B_{\mathrm{cat},n}^t A_{\mathrm{cat},n}^t = \sum_{s\in\mathcal{S}_n^t} \lambda_{s,n}^t B_{s,n}^t A_{s,n}^t, \qquad \lambda_{s,n}^t = \frac{|D_s|}{\sum_{j\in\mathcal{S}_n^t}|D_j|}.

The stacked width is Rnt=∑s∈SntrsR_n^t=\sum_{s\in\mathcal{S}_n^t}r_s, which can exceed the individual local widths.

Cross-round handling. The aggregated update is merged into the original weights: Wnt+1=Wnt+ΔWnt.W_n^{t+1}=W_n^t+\Delta W_n^t.

Layers without contributors remain unchanged. Clients synchronize to the updated backbone and initialize fresh, zero-update adapters with their prescribed widths rsr_s. Thus, the next-round training width remains rsr_s, independently of previous stacked widths. Accumulated updates are stored in the backbone's fixed-size weight matrices.

Performance and resource implications. In exact arithmetic, merging and zero-update reinitialization preserve the effective weights at the transition between rounds. Local adapter rank therefore does not grow across rounds. The main additional cost is transmitting the stacked LoRA each round. Their compact parameterization limits this overhead, and FedPruner still reduces total communication volume by up to a factor of 2.2 relative to the evaluated baselines. We will revise Appendix B and the algorithm description to specify this merge-and-reinitialize procedure and clarify its resource consumption.

W4 & Q4: Clarification of the memory profiling results.

Thanks for the constructive comment. The profiling in Figure 1 follows the OpenFedLLM[2] setup, with unquantized backbone weights stored in FP32, gradient checkpointing enabled, a batch size of 16, and a sequence length of 512.

The INT4 quantization used in our later experiments further reduces parameter storage and complements layer pruning: quantization reduces the bits stored per parameter, while pruning reduces the number of parameters retained locally. Although quantization changes the relative memory shares of different components, both techniques aim to alleviate the memory burden of base-model weights. We will explicitly document these settings and clarify the scope of Figure 1 in the revised manuscript.

Reference:

[1] FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations, 2024 NeurIPS

[2] OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learning, 2024 KDD


Finally, thanks again for your time and helpful comments. We hope your concerns are well addressed.

AuthorsOpenReview ↗

Follow-up on Our Responses

Dear Reviewer,

Thank you again for your time and constructive feedback. We have provided detailed responses to your concerns, including additional experiments and clarifications. We would be grateful if you could review our responses and let us know whether any concerns remain. We would be happy to provide further clarification.

Best regards,

The Authors

Thank you for your response and clarifications. However, given the four-point rating scale and my overall assessment of the paper's quality, I will maintain my original score.

Reviewer_py7fOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies federated LLM fine-tuning on memory-constrained devices. This is topic which has attracted lots of attention in the past few years. The paper proposes FEDPRUNER, which combines functionality-driven layer orchestration (FDLO), using CKA similarity and spectral clustering, with importance-aware layer selection. FEDPRUNER+ extends pruning to individual attention and FFN components. Experimental results show improvements over the state of the art.

Critical strengths
  1. The problem is important and timely. Combining functional layer grouping with importance-aware selection is a meaningful approach to adapting model capacity to heterogeneous devices.

  2. Comparisons with memory-aware FedRA, experiments under different memory constraints, and component ablations indicate the effectiveness of the proposed architecture, particularly for larger models.

Critical weaknesses
  1. FDLO requires client-side similarity computation over the original model before pruning. No memory-feasible execution strategy or complete resource-cost analysis is demonstrated for this step.

  2. The convergence analysis treats data-dependent partial updates as unbiased global gradients without sufficient justification. Additionally, concatenation-based LoRA aggregation may increase effective rank without an explicit control mechanism, raising scalability concerns.

  3. Although zeroth-order techniques are mentioned in the introduction and FwdLLM is evaluated as a baseline, recent advances in efficient zeroth-order federated fine-tuning have not been adequately discussed. For example, Ahmed et al., Efficient Zeroth-Order Federated Finetuning of Language Models on Resource-Constrained Devices (TMLR 2026), propose a computationally efficient solution for the same resource-constrained setting. This relevant work should be discussed in the related work section, and its exclusion from the experimental comparisons should be justified.

Questions for authors
  1. How is full-model CKA computed within a 3–9 GB device memory budget, and what are its peak memory and latency costs?

  2. What guarantees that data-dependent partial-layer updates provide unbiased global gradient estimates?

  3. How is the LoRA rank controlled after concatenation-based aggregation across devices and communication rounds?

  4. What are the independent run counts, performance variances, and end-to-end resource costs on actual edge hardware?

  5. How does FEDPRUNER compare with more recent efficient zeroth-order federated fine-tuning methods, such as Ahmed et al. (TMLR 2026)? If a direct experimental comparison is not feasible under the considered memory budgets, could the authors explain the differences and limitations of the two approaches?

4 comments
AuthorsOpenReview ↗

Response to Reviewer py7f (1/2)

We sincerely thank the reviewer for the constructive feedback. Below are our point-by-point responses to your concerns.

W1 & Q1: Computing full-model CKA under a 3–9 GB memory budget.

Thanks for the constructive comment. CKA collection does not require all original-model layers to be resident simultaneously. Appendix K (pp. 24–25) describes a pipelined workflow. To ensure the feasibility of Step 1 (Layer Similarity Computation) on resource-constrained devices, we employ a memory-efficient pipelined, layer-by-layer inference workflow. This mechanism guarantees that operations remain strictly within the device’s memory budget through a three-step process:

  • Adaptive Chunk Loading: The device loads only the initial subset of layers (e.g., layers 1-3) that fits within its current available memory.

  • Compute, Cache, and Release: A forward pass is performed on this chunk using a single data batch. Once the necessary output activations are cached and stored, the device immediately offloads the layer parameters to free memory.

  • Sequential Iteration: The device then loads the subsequent chunk (e.g., layers 4-6), utilizing the cached activations from the previous block as input. This cycle repeats until the forward pass is complete for all layers.

    This workflow ensures that the peak memory footprint during CKA calculation never exceeds the device's limit, allowing the process to adaptively scale to specific hardware constraints.

As reported in Appendix J, the calibration forward pass requires 54.3 TFLOPs, compared with approximately 2,232 TFLOPs for a local fine-tuning round. This amounts to only 2.4% of the local training computation, indicating a small additional computational overhead. Therefore, the latency introduced is also negligible.

W2 & Q2: Unbiasedness of data-dependent partial-layer updates.

Thank you for pointing this out. We agree that Assumption D.2 establishes mini-batch unbiasedness for a fixed objective, but does not guarantee that updates from data-dependent pruned architectures are unbiased estimates of the full-model gradient.

Our motivation is to allocate limited model capacity according to each client's data and memory budget. FDLO promotes coverage of different representation groups, while probabilistic IALS allows alternative layers to contribute across rounds. These choices are intended to balance resource feasibility, representational coverage, and adaptation to local data. Tables 1–3 support their empirical effectiveness, although they do not establish full-gradient unbiasedness.

We will revise Appendix D to distinguish mini-batch sampling noise from the bias introduced by architecture changes and data-dependent selection, explicitly state the additional conditions required, and narrow the convergence claim accordingly.

W2 & Q3: LoRA rank after concatenation, across clients and communication rounds.

Thank you for raising this question. The key distinction is between the stacked width used for aggregation and the local adapter width used for training. Following FLoRA[1], we merge each round's aggregated update into the backbone and reinitialize local adapters before the next round, preventing cumulative growth in trainable adapter width.

Within-round aggregation. Let Snt\mathcal{S}_n^t denote the clients retaining and updating layer nn in round tt. Each client trains a fresh adapter of width rsr_s while keeping the backbone frozen. With LoRA scaling absorbed into the factors, Equation (5) exactly represents

ΔWnt=Bcat,ntAcat,nt=∑s∈Sntλs,ntBs,ntAs,nt,λs,nt=∣Ds∣∑j∈Snt∣Dj∣.\Delta W_n^t = B_{\mathrm{cat},n}^t A_{\mathrm{cat},n}^t = \sum_{s\in\mathcal{S}_n^t} \lambda_{s,n}^t B_{s,n}^t A_{s,n}^t, \qquad \lambda_{s,n}^t = \frac{|D_s|}{\sum_{j\in\mathcal{S}_n^t}|D_j|}.

The stacked width is Rnt=∑s∈SntrsR_n^t=\sum_{s\in\mathcal{S}_n^t}r_s, which can exceed the individual local widths.

Cross-round handling. The aggregated update is then merged into the original weights: Wnt+1=Wnt+ΔWnt.W_n^{t+1}=W_n^t+\Delta W_n^t.

Layers without contributors remain unchanged. Clients synchronize to the updated backbone and initialize fresh, zero-update LoRA adapters with their prescribed widths rsr_s. Thus, the next-round training width remains rsr_s and does not inherit the stacked width RntR_n^t. The stacked factors only incur within-round aggregation and communication costs, but their width does not accumulate across rounds. We will revise Appendix B and the algorithm description to make the merge-and-reinitialize procedure explicit.

AuthorsOpenReview ↗

Response to Reviewer py7f (2/2)

Q4: Independent runs, uncertainty, and actual-edge resource costs.

Thanks for the constructive comment. We performed three runs for each experimental configuration and report the mean results. Across the evaluated models, the observed deviations from the respective means are within ±0.3% for the closed-ended benchmarks and ±0.12 for the open-ended benchmarks. We will explicitly report the run counts and define the variability measure in the revised manuscript.

We agree that actual-edge measurements are important for assessing deployment costs. As described in Section 4.8, our resource measurements were obtained on NVIDIA H800 GPUs with simulated client memory constraints. This setup enables controlled comparisons under heterogeneous memory budgets. We acknowledge that this evaluation does not fully capture the resource costs of real edge deployments. In future work, we plan to deploy FedPruner on actual edge hardware and measure its end-to-end resource consumption.

W3 & Q5: Comparison with FedSPZO.

Thank you for bringing this relevant work to our attention. FedSPZO improves the computational efficiency of multi-perturbation zeroth-order optimization through split-block computation and activation reuse.

FedSPZO optimizes how updates are estimated for the full architecture, while FedPruner customizes the architecture executed by each client through layer pruning. These approaches address complementary aspects of resource efficiency. Zeroth-order optimization reduces backpropagation-related memory requirements, but does not, by itself, reduce the number of base-model parameters. As Figure 1(c) shows under the profiled configuration, base-model weights dominate memory consumption; reducing activation storage alone therefore does not eliminate this weight-memory bottleneck.

Table A presents the additional results. FedSPZO achieves slightly higher average accuracy than FwdLLM, while FedPruner outperforms FedSPZO by 0.76 and 2.42 on TinyLLaMA and LLaMA2-7B, respectively. For LLaMA2-13B, neither FedSPZO nor FwdLLM could complete fine-tuning within the considered device-memory budgets.

Together with the memory profiling in Figure 1(c), these results support our motivation for directly reducing the local base-model footprint to accommodate memory-constrained clients. We will incorporate FedSPZO into the related-work discussion and include this comparison in the revised manuscript.

Table A: Comparison with zeroth-order optimization methods on closed-ended benchmarks.

ModelMethodTruthfulQAMMLUIFEvalBBHAvg.
TinyLLaMAFwdLLM38.5525.4918.6427.3527.51
FedSPZO38.6125.5018.7627.4027.57
FedPruner39.1626.1819.8728.0928.33
LLaMA2-7BFwdLLM45.8342.2030.5738.9739.39
FedSPZO46.1242.2530.8639.0539.57
FedPruner49.9543.6433.7240.6541.99
LLaMA2-13BFwdLLM—————
FedSPZO—————
FedPruner56.1358.0945.6248.4652.08

“—” indicates that fine-tuning could not be completed within the considered device-memory budgets under the evaluated settings.

Reference:

[1] FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations, 2024 NeurIPS


Finally, thanks again for your time and helpful comments. We hope your concerns are well addressed.

The authors' responses have addressed my concerns, and I have therefore decided to increase my score. However, I still consider the overall contribution of the work to be borderline for this venue.

AuthorsOpenReview ↗

Response to Reviewer py7f

Dear Reviewer,

Thank you for your careful reading and constructive feedback throughout the review process. We are pleased that our responses have addressed your concerns, and we sincerely appreciate your decision to increase the score.

We also understand your remaining reservations about the overall contribution. Our work connects an analysis of the parameter-memory bottleneck in on-device adaptation with a data-aware pruning framework that enables federated LLM adaptation across clients with heterogeneous memory budgets. We will clarify this contribution and incorporate the additional experimental evidence into the revised manuscript. Your feedback has helped us strengthen both the presentation and the empirical support of our work.

With sincere thanks,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.