- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors present a method called Federated Budgeted Decomposition to split a pretrained LLM into smaller sub-networks for each client in a federated learning settings, which they call sub-network specialists. This method is made up of three parts, (1) a graph-based RL policy to select sub-networks of the model relevant to a given client (2) the sub-networks are co-trained through federated learning and (3) a shared GNN modulates each sub-network. The paper evaluates the method across one model (Qwen2.5 0.5B) against independent adaptation, shape-only averaging, two hand-designed allocation rules, random structures, a full model per client and five published structured-pruning methods.
Critical strengths
-
The paper addresses a valid problem of training pruned models in a device-heterogeneous federated setting as different clients may get different pruned models and hence gradient updates may not be compatible across clients.
-
The experimental setup is well thought as the ablations are matched in both inference and training FLOPs, e.g. (1) independent adaptation for co-training, (2) shape-only averaging for alignment, (3)hand-designed rules for the allocator and random structures for the search.
Critical weaknesses
-
The paper motivates FedBUD by arguing that structured pruning “selects one sub-network… and deploys it to every client,” so each client needs its own structure chosen by a learned allocator. The paper’s own results point the other way as Hand-designed rules beat the searched structures on four of six datasets (Table 4), for example 0.8672 vs. 0.8398 on CoNLL and 0.7487 vs. 0.7269 on SQuAD. The paper itself concludes that “a hand-designed rule… can thus replace the search.” and Random structures tie the search on AG News (Table 1) and beat it on CoNLL in series A (+0.0099, Table 7). The Hypothesis 4 results show that one structure shared by every client beats per-client structures. The GNN is excluded from every reported result, and Table 8 shows differences of mixed sign within ±0.016, all from single runs. Contribution 2 lists Hypotheses 1 and 4 and “partially” Hypothesis 2, and leaves out Hypothesis 3. Yet contribution 1 and the title still center on the RL allocator, and the paper does not claim overlap averaging itself as new. Hence, the evidence supports co-training sub-networks, but not the per-client specialization or the learned allocator that the paper is built around.
-
The abstract motivates the work with “a full LLM may not even fit into one GPU, and federated fine-tuning still requires every client to run the full model.” but no memory usage is reported for any stage. The half attention action keeps “all key and value projections,” so the KV cache does not shrink. Each client also still runs the full source model on its own data. The importance scores in Eq. 3 are “computed in one forward pass over 32 training examples.” Every search candidate is sliced from the full weights and recovered with 100 LoRA steps, which takes “about one minute per candidate.” The appendix says the cost of the search is “reported separately,” but I could not find it anywhere in the paper and the paper also does not show what computational resources are available per client and hence "one minute per candidate" is not a meaningful comparision. All FLOPs are modeled rather than measured, and the only model has 0.5B parameters, which fits on any modern GPU. Hence, FedBUD keeps the requirement it criticizes in federated LoRA, that every client runs the full model, and the claim about compute-limited clients is untested.
-
All results use one source model, budget, seed, partition and three clients per federation. Specifically 3 clients is very small compared to a true federated setting. Most comparisons are single runs with no test (Tables 1, 6 and 7). Hypothesis 2 is reported as partially supported, although the text states that “no effect is established” and alignment hurts the multi-task federation in both series. The most direct baselines are also missing, (1) federated LoRA on the full model (e.g., FedIT) and (2) an off-the-shelf dense model of similar per-token FLOPs co-trained the same way. The comparison with published pruners is confounded, because FedBUD is co-trained and the pruners are not. So it does not “extend Hypothesis 1 to structures produced by other pruning methods.” Hence, the reported gains cannot be separated from seed, partition and search variability, and it is unclear whether FedBUD beats simpler alternatives.
Writing notes:
The method section is long and detailed, but most of the evidence for it is in the appendix. The main text has one results figure (Figure 2, which shows only differences) and one table (Table 1). The core results table (Table 4) appears only in the appendix. So do the comparison with hand-designed rules that refutes Hypothesis 3, all the evidence for Hypothesis 4 (Tables 5 and 7, Figure 3) and the published baselines (Table 6). The hypotheses use terms like “per-client structure,” “learned allocation” and “position-wise averaging” before they are explained. The alignment rule goes by four names: “aligned LoRA averaging,” “aggregation in the coordinates of the source model,” “coordinate alignment” and “Equation 10.” The sharing arms (“shared structure,” “pooled importance” and “another client’s structure”) first appear in the results and are never described in the method, even though Hypothesis 4 depends on them. Each results paragraph opens with “We test Hypothesis X.”, which tells the reader nothing about how or why the test was designed. Claims such as “validates” for Hypothesis 1, which is significant on only three of six datasets, overstate the evidence. Hence, the main claims cannot be checked from the main text, and the procedure behind Hypothesis 4 is not specified. A smaller issue is that Figure 1 is a raster image with no selectable text.
Questions for authors
-
What is the peak memory per client during importance scoring, search, co-training and inference, and what is the wall-clock cost of the search? Does any stage require loading the full source model on the client? What is the latency of FedBUD compared to other baselines? This matters a lot in federated settings.
-
How does FedBUD compare against federated LoRA on the full model (e.g., FedIT), and against a dense pretrained model of similar per-token FLOPs co-trained with the same federated LoRA? If the published pruners’ structures go through the same co-training, does FedBUD still score higher?
-
Hand-designed rules beat the search on four of six datasets, and a shared structure beats per-client structures. Given this, what evidence remains that each client needs its own structure? Relatedly, are the “specialists” meant to be personalized models? After each round, clients receive and (Eq. 10), so their adapters are overwritten by averages. Is evaluation done before or after the final aggregation?
-
How are the sharing arms constructed, and how many were compared per task on validation data? Why did validation select another client’s structure on SQuAD when pooled importance scored higher?