- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies federated LLM fine-tuning under heterogeneous memory constraints and proposes dynamic layer pruning to construct submodels tailored to client resources. Its macro-level module, FDLO, constructs a graph using CKA similarities between layer outputs on local data and partitions the layers into as many groups as the client can retain layers. Its micro-level module, IALS, derives within-group sampling probabilities from differences between layer inputs and outputs, selecting one representative layer per group. The resulting submodels undergo local LoRA fine-tuning, after which updates are uploaded for aggregation. The fine-grained variant, FedPruner+, extends pruning to individual MHA and FFN components. Through instruction tuning on Alpaca-GPT4, closed-ended and open-ended evaluations, memory-budget experiments, module ablations, and resource measurements, the paper reports improvements in model performance, client participation, and training efficiency.
Critical strengths
S1. The problem is well defined, and the method directly addresses the memory burden of base-model weights. The paper clearly distinguishes reducing the number of trainable parameters from reducing the memory required to keep the model resident. Its profiling demonstrates that, under the tested configurations, optimizing LoRA modules alone does not eliminate the memory overhead of the base-model weights. FedPruner instead adapts the model structure used in each client’s forward and backward computations, allowing resource-constrained clients to contribute their local data. The proposed structured pruning approach directly addresses the identified bottleneck and has clear practical relevance to federated fine-tuning under heterogeneous resource constraints.
S2. Comparisons with FedRA show that the gains cannot be attributed solely to enabling more clients to participate. In Table 1, both FedRA and FedPruner allow all devices to participate, yet FedPruner improves average closed-ended benchmark performance by 1.71 and 4.30 percentage points on LLaMA2-7B and LLaMA2-13B, respectively. Table 2 further compares the methods under matched budgets expressed as fractions of the global model. For example, under the 25% budget for the 13B model, FedPruner achieves 50.56 compared with FedRA’s 44.99. These comparisons help isolate the value of the submodel construction strategy and provide key evidence for the methodological contribution.
S3. Ablation studies directly support the contributions of macro-level grouping and micro-level selection. Table 3 removes FDLO, IALS, and both modules. On LLaMA2-13B, average performance decreases by 2.85, 1.29, and 4.30 percentage points, respectively, with consistent trends on the 7B model. These results support contributions from both modules rather than gains driven by only one component. In addition, Table 4 reports that FedPruner reduces training time from 2.10 to 1.92 hours and communication volume from 3.65 to 3.47 GB relative to FedRA, indicating that its accuracy improvements do not come at the expense of higher time or communication costs in this experiment.
Critical weaknesses
W1. Direct evidence for the explanation of the pruning mechanism remains limited. The method groups layers according to output similarity and selects a representative layer from each group to preserve hierarchical information-processing capabilities. Figure 6 in Section 4.4 shows a selection pattern favoring early and late layers, while the ablations in Table 3 support the contributions of both modules to final performance. These findings support the effectiveness of the selection strategy, but do not directly explain why grouping by representational similarity preserves essential hierarchical processing capabilities under aggressive pruning.
W2. The additional benefits of client-specific data adaptation have not been clearly isolated. Section 3.1 presents local similarity computation as a mechanism for handling data heterogeneity, allowing clients to construct submodels according to their own data distributions. The overall results and memory-budget experiments in the main paper support the framework’s effectiveness under resource constraints. However, they do not clearly distinguish gains from generally better layer selection from those attributable to adaptation to individual clients’ data distributions.
Questions for authors
Q1. Under aggressive pruning, to what extent does FedPruner’s advantage come from constructing submodels that better preserve the original model’s capabilities, versus subsequent dynamic training compensating for pruning-induced degradation? Could the authors draw on existing submodel performance results to further explain why grouping layers by similarity and retaining one representative per group preserves important inter-layer interactions?
Q2. How does FedPruner’s advantage over FedRA change as the data distributions across clients become more heterogeneous?
Response to Reviewer zmhQ
We sincerely thank the reviewer for the constructive and positive feedback. Below are our point-by-point responses to your concerns.
Thank you for this insightful question. Our rationale is that individual-layer redundancy does not imply that multiple layers can be safely removed together. Several layers may provide overlapping representations, making each appear individually dispensable; removing them jointly can eliminate an intermediate processing stage needed by subsequent layers. Figures 2 and 3 motivate this concern: limited degradation from individual-layer removal does not translate into reliable performance under multi-layer pruning.
FDLO addresses this joint-selection problem by grouping layers according to output similarity on local data and retaining one representative from each group. This reduces redundancy within similar regions while preventing any identified representation group from being omitted entirely. With representatives retained in their original order, the submodel maintains a compressed processing path across different representation stages, helping preserve the composition of earlier-stage representations and later-stage transformations. IALS then favors higher-scoring representatives within each group.
Appendix G illustrates this allocation. In Table 6, the 32-layer model retains layers 1–5 and 27–32 as singleton representatives, together with one representative from the highly similar middle region, layers 6–26. The ablations in Table A further support the complementary roles of the two components. Removing FDLO decreases average accuracy by 1.22 and 2.85 on LLaMA2-7B and LLaMA2-13B, respectively. The corresponding decreases are 0.57 and 1.29 without IALS, and 1.71 and 4.30 without both components. The largest decline occurs when both are removed, providing further evidence for the effectiveness of our coordinated submodel construction strategy.
Table A: Ablation analysis of FedPruner.
Thank you for this constructive comment. To directly examine how the advantage over FedRA changes with data heterogeneity, we conducted additional experiments on YELP-P and AGNEWS, using RoBERTa as the global model. These benchmarks provide explicit class labels, allowing us to construct three client-data distribution settings: IID, Non-IID, and Severe Non-IID.
As shown in Table B, FedPruner consistently outperforms FedRA, and its advantage widens as heterogeneity increases. On YELP-P, the accuracy gap grows from 8.33 under IID to 10.48 under Non-IID and 14.11 under Severe Non-IID. On AGNEWS, the corresponding gains increase from 8.55 to 11.89 and 16.13. These results provide empirical evidence that FedPruner better accommodates increasing client-data heterogeneity. We will include these additional results in the revised manuscript.
Table B: Accuracy under different client-data distributions.
Finally, thanks again for your time and helpful comments. We hope your concerns are well addressed.