acceptodds
Withdrawn from ICLR 2027

Budget-Constrained Decomposition of LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Adapting large language models (LLMs) is highly resource-demanding: for low-resource research institutions, a full LLM may not even fit into one GPU, and federated fine-tuning still requires every client to run the full model. Existing remedies fall short. First, structured pruning selects one sub-network on a generic calibration corpus and deploys it to every client. Second, federated methods that derive per-client sub-networks bound parameters or memory, not per-token compute. To overcome this, we propose Federated Budgeted Decomposition (FedBUD), which decomposes one pretrained LLM into smaller per-client structural specialists under a hard per-token FLOPs budget. A graph-based reinforcement-learning policy selects each client's blocks, attention heads and feed-forward neurons from its architecture graph and importance statistics. The specialists are then co-trained via federated learning, with LoRA factors aggregated in the coordinates of the source model; an optional shared graph neural network modulates each specialist by its architecture graph, with representation discrepancy bounded by architecture-graph distance for its attention instantiation. FLOPs-matched experiments on six federations show that co-training outperforms independent adaptation across the six datasets, significantly on three, and that sharing structure across clients, selected on validation data, significantly improves both tasks where the selected arm was run at three seeds, whereas the learned allocator does not outperform hand-designed allocation rules. Compute-limited clients can thus co-train LLM specialists without pooling data. Code will be released upon acceptance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
structured pruning, federated learning, model heterogeneity, large language models, parameter-efficient fine-tuning, graph neural networks, FLOPs-matched evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KHwHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors present a method called Federated Budgeted Decomposition to split a pretrained LLM into smaller sub-networks for each client in a federated learning settings, which they call sub-network specialists. This method is made up of three parts, (1) a graph-based RL policy to select sub-networks of the model relevant to a given client (2) the sub-networks are co-trained through federated learning and (3) a shared GNN modulates each sub-network. The paper evaluates the method across one model (Qwen2.5 0.5B) against independent adaptation, shape-only averaging, two hand-designed allocation rules, random structures, a full model per client and five published structured-pruning methods.

Critical strengths
  1. The paper addresses a valid problem of training pruned models in a device-heterogeneous federated setting as different clients may get different pruned models and hence gradient updates may not be compatible across clients.

  2. The experimental setup is well thought as the ablations are matched in both inference and training FLOPs, e.g. (1) independent adaptation for co-training, (2) shape-only averaging for alignment, (3)hand-designed rules for the allocator and random structures for the search.

Critical weaknesses
  1. The paper motivates FedBUD by arguing that structured pruning “selects one sub-network… and deploys it to every client,” so each client needs its own structure chosen by a learned allocator. The paper’s own results point the other way as Hand-designed rules beat the searched structures on four of six datasets (Table 4), for example 0.8672 vs. 0.8398 on CoNLL and 0.7487 vs. 0.7269 on SQuAD. The paper itself concludes that “a hand-designed rule… can thus replace the search.” and Random structures tie the search on AG News (Table 1) and beat it on CoNLL in series A (+0.0099, Table 7). The Hypothesis 4 results show that one structure shared by every client beats per-client structures. The GNN is excluded from every reported result, and Table 8 shows differences of mixed sign within ±0.016, all from single runs. Contribution 2 lists Hypotheses 1 and 4 and “partially” Hypothesis 2, and leaves out Hypothesis 3. Yet contribution 1 and the title still center on the RL allocator, and the paper does not claim overlap averaging itself as new. Hence, the evidence supports co-training sub-networks, but not the per-client specialization or the learned allocator that the paper is built around.

  2. The abstract motivates the work with “a full LLM may not even fit into one GPU, and federated fine-tuning still requires every client to run the full model.” but no memory usage is reported for any stage. The half attention action keeps “all key and value projections,” so the KV cache does not shrink. Each client also still runs the full source model on its own data. The importance scores in Eq. 3 are “computed in one forward pass over 32 training examples.” Every search candidate is sliced from the full weights and recovered with 100 LoRA steps, which takes “about one minute per candidate.” The appendix says the cost of the search is “reported separately,” but I could not find it anywhere in the paper and the paper also does not show what computational resources are available per client and hence "one minute per candidate" is not a meaningful comparision. All FLOPs are modeled rather than measured, and the only model has 0.5B parameters, which fits on any modern GPU. Hence, FedBUD keeps the requirement it criticizes in federated LoRA, that every client runs the full model, and the claim about compute-limited clients is untested.

  3. All results use one source model, budget, seed, partition and three clients per federation. Specifically 3 clients is very small compared to a true federated setting. Most comparisons are single runs with no test (Tables 1, 6 and 7). Hypothesis 2 is reported as partially supported, although the text states that “no effect is established” and alignment hurts the multi-task federation in both series. The most direct baselines are also missing, (1) federated LoRA on the full model (e.g., FedIT) and (2) an off-the-shelf dense model of similar per-token FLOPs co-trained the same way. The comparison with published pruners is confounded, because FedBUD is co-trained and the pruners are not. So it does not “extend Hypothesis 1 to structures produced by other pruning methods.” Hence, the reported gains cannot be separated from seed, partition and search variability, and it is unclear whether FedBUD beats simpler alternatives.


Writing notes:

The method section is long and detailed, but most of the evidence for it is in the appendix. The main text has one results figure (Figure 2, which shows only differences) and one table (Table 1). The core results table (Table 4) appears only in the appendix. So do the comparison with hand-designed rules that refutes Hypothesis 3, all the evidence for Hypothesis 4 (Tables 5 and 7, Figure 3) and the published baselines (Table 6). The hypotheses use terms like “per-client structure,” “learned allocation” and “position-wise averaging” before they are explained. The alignment rule goes by four names: “aligned LoRA averaging,” “aggregation in the coordinates of the source model,” “coordinate alignment” and “Equation 10.” The sharing arms (“shared structure,” “pooled importance” and “another client’s structure”) first appear in the results and are never described in the method, even though Hypothesis 4 depends on them. Each results paragraph opens with “We test Hypothesis X.”, which tells the reader nothing about how or why the test was designed. Claims such as “validates” for Hypothesis 1, which is significant on only three of six datasets, overstate the evidence. Hence, the main claims cannot be checked from the main text, and the procedure behind Hypothesis 4 is not specified. A smaller issue is that Figure 1 is a raster image with no selectable text.

Questions for authors
  1. What is the peak memory per client during importance scoring, search, co-training and inference, and what is the wall-clock cost of the search? Does any stage require loading the full source model on the client? What is the latency of FedBUD compared to other baselines? This matters a lot in federated settings.

  2. How does FedBUD compare against federated LoRA on the full model (e.g., FedIT), and against a dense pretrained model of similar per-token FLOPs co-trained with the same federated LoRA? If the published pruners’ structures go through the same co-training, does FedBUD still score higher?

  3. Hand-designed rules beat the search on four of six datasets, and a shared structure beats per-client structures. Given this, what evidence remains that each client needs its own structure? Relatedly, are the “specialists” meant to be personalized models? After each round, clients receive Bˉ[Kk,:]\bar{B}[K_k,:] and Aˉ\bar{A} (Eq. 10), so their adapters are overwritten by averages. Is evaluation done before or after the final aggregation?

  4. How are the sharing arms constructed, and how many were compared per task on validation data? Why did validation select another client’s structure on SQuAD when pooled importance scored higher?

Reviewer_SJS7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper tackles adapting LLMs in a federated environment, subject to the computational resources (FLOPs, memory) that each client can provide for the training, which may limit using the full model. To this end, it proposes an FL algorithm to train per-client specialized models extracted from a common, pretrained LLM, without access to previously collected centralized data. The proposed method works involves instead a local importance-based scoring of parameters based on private client data, an allocator that selects the structure and a federated training for LoRA adapters, added to recover the error of pruning weights. Experiments on a single Qwen2.5-0.5B model on five single-task federations indicate mild gains derived from co-training.

Critical strengths

I genuinely cannot identify any strong strengh of this work, since the motivation and research questions appear to me already questionable from the beginning, and the provided empirical evidence does not substantiate significant gains justifying the overall methodology.

In terms of findings, or useful results that the community may be interested in, there is the empirical evidence that co-training outperforms independent adaptation ("Hypothesis 1"); however the experimentation is too narrow to justify this claim more broadly.

Critical weaknesses

There are several critical points right at the premise level of the work, which consequently affect the rest of the paper.

1. Motivation The research question of the work lies on two premises: (i) a full LLM may not fit into the memory of one GPU, meaning that extracting a subnetwork should not require evaluating the full model, and (ii) existing methods for heterogeneous model in FL size each sub-model by device capacity (I assume the authors mean "memory") or by a parameter count, creating a mismatch between the selection criterion and the actual constraints at training/deployment.

On both points, the paper does not show a convincing solution. About subnetwork extraction, importance estimation requires a forward pass of the source model on private client examples: while offloading or sequential execution could mitigate, the paper does not provide a really new solution. On the claim that parameter count can be a misleading metric, this is in general true, as attention term depends on context length and retained query heads, but at fixed context length, their FLOPs objective remains closely tied to retained parameter count. More specifically I see:

  1. No measured latency, throughput, or peak memory is reported, the paper uses analytical FLOPs at a fixed context length of 400. FLOPs themselves are only a proxy for deployment latency, so the claim should be restricted as consequence.
  2. No controlled comparison of parameter-constrained versus FLOPs-constrained allocation with everything else kept the same
  3. No strong evidence that existing approaches could not be calibrated on a compute metric target, matching the main claim in abstact/introduction.

Beside this, there is one question that remains unanswered: why should each client need a different architecture, as an alternative to use a common, good one (constrained on compute/memory) and just learn its adapters? In other words, I don't see why the mapping between budget (however calculated) and retained params should differ across clients, and so the necessity of the full machinery presented in the paper.

The motivation of the paper seems closer to "one pretrained model, multiple deployment budgets" plus adaptation to client distribution. In this sense, methods from elastic training literature (e.g. FlexTron, LLamaFlex and FlexRank) would be natural baseline to extract budget-compatible elastic subnetworks and federate their adapters. Their shared component ordering would also simplify correspondence between clients. FedBUD neither evaluates these alternatives nor demonstrates that its non-nested, client-specific selection is preferable.

2. The results do not justify the method’s complexity

The experiments show gains from co-training over independently adapting the same subnetworks, but do not establish the benefit of the proposed machinery over simpler federated baselines. The learned allocator loses to simple rules on four of six datasets. Coordinate alignment fixes component correspondence, but averaging LoRA factors still does not average the effective updates; the authors propose a correction but never test it. Finally, the theorem concerns an optional GNN excluded from the main results and does not explain why co-training should improve performance.

These findings leave unclear what the full method adds over simpler pruning from related works followed by federated adaptation.

Questions for authors

My concerns concern the motivation and the evidence needed to support the central claims, rather than ambiguities that clarification could resolve. Addressing them would require substantial additional experiments, so I have no specific questions for the rebuttal.

Reviewer_WaB5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FedBUD to build smaller client models from one pretrained LLM under a per-token FLOPs budget. It selects blocks, attention heads, and feed-forward neurons using local data. Then, it trains the models using LoRA. The paper also studies a shared GNN and gives a stability bound. Results show that co-training helps compared with independent training.

Critical strengths
  1. This paper is well-organized, well-written, and easy to follow.
  2. The motivation is valid. Building smaller models for clients with limited resources and allowing them to learn together has a clear and practical value.
  3. The main co-training experiments use the same structures, training steps, and paired seeds for a fair comparison. Co-training improves the mean result on all six federations, with significant gains on three. The SQuAD gain of F1 score is substantial as well.
Critical weaknesses
  1. The main results are spread across two sets of experiments and several tables with different numbers of runs. The most useful three-seed comparison is in Appendix Table 4. It will be better if the author reorganizes the paper to magnify the contribution and significance of the work.
  2. Compared with previous works that deal with graph-based reinforcement learning, sub-model training with shared parameter averaging, and federated LLM pruning, FedBUD is an extension to address structurally different LLMs. It is important that the authors highlight the contribution and uniqueness of the proposed framework.
  3. The connection between GNN and the proposed framework is still unclear. The results of GNN in Table 8 show both good and bad results. Its stability bound is with respect to a module, while the whole GNN setting is not used in the main experiments, so it does not explain their main gains.
  4. The experimental coverage is limited. The authors only use Qwen2.5-0.5B, which is both outdated and small. Three/four clients with one fixed budget do not scale. In addition, alignment would degrade the multi-task result in both experimental series. This may limit the benefit when clients have different tasks.
Questions for authors
  1. What are the detailed experimental settings (e.g., earning rate, batch size, warmup, and training sequence length)?
  2. Can authors explain clearly the contribution of the GNN, instead of marking it as optional or an add-on?
  3. Why does alignment degrade the performance of the multi-task setting?
  4. What is the significance of Theorem 4.1, and how may it affect the convergence?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.