acceptodds
Under review as a conference paper at ICLR 2027

Who Trains What? Disentangled Global Model Modularization for Federated Learning Guided by Client Metadata

Abstract

Federated Learning (FL) benefits from numerous client participations, yet resource-constrained clients with sufficient data may be excluded due to straggler effects. We study which parts should each client train given its local data and computation budget? This is difficult because the same model structure in a global model produces overlapping responses to different data characteristics, while one data characteristic may involve many structures. We propose MDMP, a metadata-guided method for disentangled global model partitioning. MDMP measures the responses of model structures to different data characteristics. It then groups structures with similar responses into functional modules and selects for each client a set that covers its major data characteristics within its training budget. Experiments on MNIST, CelebA, and CIFAR-10 show that, under a 95% target computation budget, MDMP reduces measured computation by 6.38–9.24% with at most a 0.94 percentage-point drop in mean client accuracy.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Federated Learning, Model partition, Data characteristics

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_8sQpOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies model partition in resource- and data-heterogeneous federated learning and proposes MDMP, a metadata-guided post-training partitioning method for heterogeneous federated learning. MDMP calculates condition gradients of structural units and calibrates a global model for response concentration and separation to group those units into different modules. Then, the considered model partition problem was reformulated to solve a metadata-aware set-cover problem for each client. Experiments on 10-client MNIST, CelebA, and CIFAR-10 evaluate measured MACs, mean client accuracy, worst-client drops, and stability. At the 95% target budget, MDMP has only 6.38-9.24% computation savings with at most a 0.94 percentage-point mean accuracy drop, while the maximum client drop remains below the stated 5-point safety limit.

Critical strengths
  1. The paper addresses an important gap in heterogeneous federated learning. Whereas existing methods primarily adapt sub-model size to client capacity, MDMP introduces a data-aware perspective that jointly considers clients’ computational budgets and the data conditions they need to learn. This formulation is practically meaningful because enabling resource-constrained clients to participate can improve participation diversity and overall model utility.
  2. MDMP develops a coherent methodological pipeline that connects condition-gradient analysis, capacity-balanced module construction, and client-specific module selection. This design systematically translates client-side data and resource heterogeneity into executable personalized sub-networks, rather than treating model-size adaptation as an isolated pruning problem.
  3. The experiments provide strong deployment-oriented evidence across multiple forms of heterogeneity, including label and morphological variation on MNIST, attribute-based feature heterogeneity on CelebA, and CLIP-derived visual conditions on CIFAR-10. The evaluation reports executable MACs, mean client accuracy, per-client degradation, and worst-client performance drop. Measuring actual sub-network computation—rather than relying only on retained parameter ratios—directly supports the paper’s efficiency claims.
  4. The module-removal test further verifies whether the learned modules capture their intended data conditions, while supplementary analyses examine client-level retention, module granularity, resource usage, and selection–test consistency. Together, these analyses provide useful mechanistic and robustness evidence beyond aggregate accuracy comparisons.
Critical weaknesses
  1. The central concentration objective appears mathematically inconsistent with its stated purpose. Equation (2) defines LconL_{con} as negative normalized entropy, while LcalL_{cal} is minimized. Therefore, minimizing the displayed objective appears to encourage high-entropy, diffuse responses rather than concentrated ones. The authors should clarify the exact implementation and indicate whether the sign in Equation (2) is a typographical error.
  2. The scalability of the method has not been sufficiently established. As the numbers of structural units, metadata conditions, and clients increase, the associated computational and communication overheads may grow substantially. Experiments limited to 10 clients and ResNet-18 are insufficient to demonstrate that MDMP remains practical for larger models and larger-scale federated learning systems.
  3. The assumption that metadata-conditioned gradient responses reveal functional modularity is not sufficiently justified. Although the simulation provides some evidence of gradient-response “disentanglement,” the paper does not clearly explain why such patterns should correspond to meaningful functional modularity in actual neural networks. A stronger theoretical motivation or empirical validation of this connection is needed.
  4. The algorithm is difficult to reproduce because several key definitions and implementation details are underspecified. The paper introduces numerous symbols and multiple optimization stages without providing a concise and unified technical description. A clear algorithmic summary, together with precise definitions of the main variables, objectives, and optimization steps, would substantially improve reproducibility.
Questions for authors
  1. Equation (2) defines LconL_{con}, while LcalL_{cal} is minimized. For a binary condition, a uniform response gives Lcon=−1L_{con}=-1, whereas a one-hot response gives Lcon=0L_{con}=0. Thus, minimizing the displayed objective appears to maximize entropy rather than concentrate responses. Is the sign in Eq. (2) incorrect, or is a maximization step used in the implementation?
  2. Since the proposed MDMP is based on clients’ metadata, it is unclear how they can get such information. Please clarify how metadata is obtained at clients in practical deployment, and indicate which metadata are assumed available without an additional recognition model.
  3. The primary feature set is described as selecting “at most” smaxs_{max}. If all responses are zero, or if several conditions have identical responses, the set Smx\mathcal{S}_{mx} may be empty. How are empty sets handled in implementation?
  4. Since there exist numerous mathematical symbols in the paper, which causes the method MDMP hard to follow, could the authors provide an overall technical scheme for MDMP?
  5. The number of modules MM, response threshold τresp\tau_{resp}, feature-selection limit smaxs_{max}, coverage parameter ρjℓ\rho_j^\ell, and capacity factor α\alpha can all affect the final sub-networks. The paper provides an ablation on module-count for CelebA, but it is unclear whether the conclusions are stable on MNIST and CIFAR-10. A sensitivity study or ablation table would help separate robust behavior from dataset-specific tuning.
5 comments
AuthorsOpenReview ↗

Response (Part 1/3)

Thank you to the reviewers for your valuable feedback on our paper. We sincerely thank you for taking the time to review our work, and we are grateful for your insightful comments.In this rebuttal, we address the key concerns raised and seek to clarify the relevant issues while incorporating the suggestions to improve our paper.

Weakness 1 & Question 1: The central concentration objective appears mathematically inconsistent with its stated purpose. Equation (2) defines Lcon\mathcal{L}_{\mathrm{con}} as negative normalized entropy, while Lcal\mathcal{L}_{\mathrm{cal}} is minimized. Therefore, minimizing the displayed objective appears to encourage high-entropy, diffuse responses rather than concentrated ones. The authors should clarify the exact implementation and indicate whether the sign in Equation (2) is a typographical error.

R: Thank you for raising this question. We verified Eq. (2): the outer minus sign makes the expression normalized entropy, and minimizing it promotes response concentration. The implementation directly minimizes the calibration objective containing this entropy term.

For structural unit qq and single-valued factor ℓ\ell, the normalized response entropy is Hqℓ=−∑j∈Jℓϕq,jℓ,callog⁡ϕq,jℓ,callog⁡∣Jℓ∣H_{q}^{\ell} = -\frac{\sum_{j \in \mathcal{J}_{\ell}} \phi_{q,j}^{\ell,\mathrm{cal}} \log \phi_{q,j}^{\ell,\mathrm{cal}}}{\log |\mathcal{J}_{\ell}|} We have $\mathcal{L}_{\mathrm{con}}^{\mathrm{cal}} =\frac{1}{Q|\mathcal{I}_s|}\sum_{q=1}^{Q}\sum_{\ell\in\mathcal{I}_s}H_q^\ell$ after taking its average to Eq (2). Since the uniform response ϕ=(1/2,1/2)\boldsymbol{\phi}=(1/2,1/2), we have

Hqℓ=−12log⁡12+12log⁡12log⁡2=1,H_q^\ell=-\frac{\frac12\log\frac12+\frac12\log\frac12}{\log 2}=1, Then, the one-hot response ϕ=(1,0)\boldsymbol{\phi}=(1,0) is Hqℓ=−1log⁡1+0log⁡0log⁡2=0H_q^\ell=-\frac{1\log 1+0\log 0}{\log 2}=0. Therefore, −1-1 and 00 are the results when the outermost negative sign is not taken into account. According to Eq (2), they should be 11 and 00 respectively when calculated completely. Since Lconcal\mathcal{L}_{\mathrm{con}}^{\mathrm{cal}} is included in Lcal\mathcal{L}_{\mathrm{cal}} with a positive weight, minimizing the concentration term in this objective will encourage more concentrated conditional responses.

Weakness 2: The scalability of the method has not been sufficiently established. As the numbers of structural units, metadata conditions, and clients increase, the associated computational and communication overheads may grow substantially. Experiments limited to 10 clients and ResNet-18 are insufficient to demonstrate that MDMP remains practical for larger models and larger-scale federated learning systems.

R: Thank you for suggesting that we add more extensive validation. On the MNIST, CelebA and CIFAR-10 datasets, we increased the number of clients from 10 to 20 and re-executed the entire process of MDMP.

In Table 1, Global and MDMP represent the average test accuracy of the calibrated global model and the assigned sub-models respectively, Δ\Delta is the difference between the two, and Feas is the number of clients that meet the accuracy and MACs budget. Under the 90%90\% MACs budget for MNIST, one client uses the complete model, and its accuracy is included in the average value, but it is not counted as feasible within the MACs budget.

Table 1. MDMP with 20 clients. \begin{array}{lcrrrc}\text{Dataset} & \text{MACs Budget} & \text{Global (\\%)} & \text{MDMP (\\%)} & \Delta & \text{Feas} \\\\ \hline \text{MNIST} & 95\\% & 99.69 & 99.23 & 0.47 & 20/20 \\\\ \hline \text{MNIST} & 90\\% & 99.69 & 97.09 & 2.61 & 19/20 \\\\ \hline \text{CelebA} & 95\\% & 92.51 & 92.50 & 0.01 & 20/20 \\\\ \hline \text{CelebA} & 90\\% & 92.51 & 92.50 & 0.01 & 20/20 \\\\ \hline \text{CIFAR-10} & 95\\% & 83.73 & 82.59 & 1.14 & 20/20 \\\\ \hline \text{CIFAR-10} & 90\\% & 83.73 & 81.20 & 2.53 & 20/20\end{array}

The results show that under a 95%95\% MACs budget, all 20 clients in each of the three datasets obtained feasible sub-models, with the average accuracy decreasing by only 0.470.47, 0.010.01, and 1.141.14 respectively; under a 90%90\% MACs budget, 19/2019/20, 20/2020/20, and 20/2020/20 clients were feasible respectively, and the average accuracy decreased by no more than 2.612.61. This indicates that even when the number of clients increases to 20, MDMP remains practical.

Weakness 3: The assumption that metadata-conditioned gradient responses reveal functional modularity is not sufficiently justified. Although the simulation provides some evidence of gradient-response “disentanglement,” the paper does not clearly explain why such patterns should correspond to meaningful functional modularity in actual neural networks. A stronger theoretical motivation or empirical validation of this connection is needed.

AuthorsOpenReview ↗

Response (Part 2/3)

R:** Thank you for the suggestion. The module construction of MDMP is based on condition loss sensitivity, response similarity, and module removal verification. BIMT [1] demonstrates instances where neural networks, under appropriate training constraints, form interpretable modules, share features, and collaborate for prediction, providing empirical evidence for organizing modules based on internal computational functions.

Sensitivity and grouping basis: Appendix FF indicates that under the same small perturbation budget, the maximum first-order change of the conditional loss is ε∥∇θqFc∥2\varepsilon\|\nabla_{\theta_q}F_c\|_2 . Therefore, the normalized conditional gradient can describe the local sensitivity of the structural unit; Taylor pruning [2] and Task2Vec [3] also study the relationship between tasks and structures by utilizing the derivatives of losses. For unit vectors, maximizing the cosine similarity is equivalent to minimizing the squared distance (Appendix C.2C.2), enabling the structurally similar conditional responses to form functional modules.

Prediction contribution verification: Appendix HH further compares the conditional losses before and after the removal of the module. If removing a module leads to an increase in the loss under condition cc , it indicates that when other modules are retained, this module is helpful for the prediction of that condition.This links the local sensitivity described by the gradient with the actual predicted changes.

[1] Liu et al. Seeing Is Believing: Brain-Inspired Modular Training for Mechanistic Interpretability. Entropy, 2024.

[2] Molchanov et al. Pruning Convolutional Neural Networks for Resource Efficient Inference. ICLR, 2017.

[3] Achille et al. Task2Vec: Task Embedding for Meta-Learning. ICCV, 2019.

Weakness 4 & Question 4: The algorithm is difficult to reproduce because several key definitions and implementation details are underspecified. The paper introduces numerous symbols and multiple optimization stages without providing a concise and unified technical description. A clear algorithmic summary, together with precise definitions of the main variables, objectives, and optimization steps, would substantially improve reproducibility. Could the authors provide an overall technical scheme for MDMP?

R: Thank you for the suggestion. To reduce the cognitive burden caused by mathematical symbols, we have provided the overall framework diagram and corresponding explanations in Appendix JJ. The MDMP consists of three parts:

(a) functional module construction, which starts from a pretrained federated global model and performs response calibration, conditional-response measurement, and module partitioning in advance; (b) Client sub-module determination, which selects modules from the existing library based on client's metadata distribution and MACs budget; (c) Participation by resource-constrained clients, which assigns submodels that satisfies MACs budget to clients, enabling local model training. Note that in the above three steps, step (a) and (b) should be "trained" on some clients in advance before later deployment in FL to allow resource-constrained clients to participant.

Question 2: Since the proposed MDMP is based on clients’ metadata, it is unclear how they can get such information. Please clarify how metadata is obtained at clients in practical deployment, and indicate which metadata are assumed available without an additional recognition model.

R: Thank you for the suggestion. This paper primarily utilizes the existing labeled or directly calculable attributes. When such attributes are unavailable, they are estimated using the model.CelebA directly reads the official attribute annotations and uses 39 attributes other than the task objective "Smiling" as metadata; MNIST uses six morphological measurements from Morpho-MNIST as metadata, and new samples can be calculated through the corresponding algorithm.CIFAR-10 reuses the eight attribute word list made public by LM4CV. It calculates the matching scores for each image through the frozen CLIP, and then constructs high/low conditions based on the threshold determined by the training set. Therefore, it requires additional model inference.

For new datasets lacking pre-existing attributes, a process based on the LM4CV approach can be adopted: LLM generates candidate attributes based on the category description → freeze CLIP to extract text-image features → learn K attribute directions using the training set category labels, match the candidate text and filter the attributes → calculate the matching scores for each image and discretize them into metadata.

Question 3: The primary feature set is described as selecting “at most smax⁡s_{\max} . If all responses are zero, or if several conditions have identical responses, the set SmxS_{mx} may be empty. How are empty sets handled in implementation?

AuthorsOpenReview ↗

Response (Part 3/3)

R: Thank you for the question. The main condition set SmS_m is selected in descending order of module relative response em,jℓe_{m,j}^{\ell}. "At most smax⁡s_{\max}" specifies the maximum set size. Sort all candidate conditions in descending order of response, and handle zero responses and tied responses using a fixed index sequence.

Specifically, stable sorting preserves the predefined condition order when responses are equal and returns min⁡(smax⁡,V)\min(s_{\max},V) conditions. Here, VV is the total number of candidate conditions, and the code parameter major_max corresponds to smax⁡s_{\max}. If all input responses are zero, the fixed index order determines the condition labels. The corresponding code can be found in the file "mnist_resnet_modules_code/mnist_resnet_modules/modules.py" in the directory "MNIST_experiments", specifically on lines 134 and 140.

Before module construction, a numerical validity check is also performed: if the maximum row norm of the response matrix does not exceed 10−1110^{-11}, the code will report an error and terminate the process. The corresponding code is located in "mnist_resnet_modules_code/mnist_resnet_modules/modules.py" in the directory "MNIST_experiments", specifically on lines 100 and 101.

Question 5: The number of modules MM, response threshold τresp\tau_{\mathrm{resp}}, feature-selection limit smax⁡s_{\max}, coverage parameter ρjℓ\rho_j^{\ell}, and capacity factor α\alpha can all affect the final sub-networks. The paper provides an ablation on module-count for CelebA, but it is unclear whether the conclusions are stable on MNIST and CIFAR-10. A sensitivity study or ablation table would help separate robust behavior from dataset-specific tuning.

R: Thank you for the suggestion. We conducted additional experiments on the number of modules for MNIST and CIFAR-10, as well as a sensitivity analysis of the remaining parameters on MNIST.

Module count MM: Table 2 compares the results of MNIST and CIFAR-10 under different numbers of modules and two levels of MACs Budget. Acc. represents the average test accuracy range of the client, and All-client feasible is the number of configurations where all clients meet the budget constraint divided by the total number of configurations.

Table 2. Module-count sensitivity. \begin{array}{llcrr}\text{Dataset} & M & \text{MACs Budget} & \text{Acc. (\\%)} & \text{All-client feasible} \\\\ \hline \text{MNIST} & 4,6,10,12 & 95\\% & 97.57\text{–}99.44 & 4/4 \\\\ \hline \text{MNIST} & 4,6,10,12 & 90\\% & 97.03\text{–}98.90 & 3/4 \\\\ \hline \text{CIFAR-10} & 12,14,16,18 & 95\\% & 86.80\text{–}88.13 & 4/4 \\\\ \hline \text{CIFAR-10} & 12,14,16,18 & 90\\% & 85.38\text{–}87.00 & 4/4 \\\\ \end{array}

Among the 16 new configurations, 15 achieve full client budget-based allocation, with the average accuracy drop not exceeding 3.143.14. It supports effective accuracy at different module granularities - a trade-off between computational cost and accuracy. The complete results can be found in Appendix I.9 Table 77.

Other parameters:** Table 3: Fixed M=8M=8 on MNIST, default τresp=10−6\tau_{\mathrm{resp}}=10^{-6}, c=0.5c=0.5, and α=2\alpha=2. . Change the parameters item by item. Values represent the parameter values, Acc. indicates the average test accuracy range under the corresponding MACs budget, Feasible reports the number of feasible clients in order of values, and there are 10 clients in each group.

**Table 3. Parameter sensitivity on MNIST. \begin{array}{llrrl}\text{Parameter} & \text{Values} & \text{Acc (95\\%)} & \text{Acc (90\\%)} & \text{Feasible (90\\%)} \\\\ \hline \tau_{\mathrm{resp}} & 10^{-6},0.01,0.05 & 97.46\text{–}99.02 & 96.46\text{–}98.70 & 10,9,2 \\\\ \hline \rho_j^{\ell}=c/J_{\ell} & c=0.25,0.5,0.75 & 98.76\text{–}99.48 & 96.98\text{–}97.73 & 10,10,5 \\\\ \hline \alpha & 1.5,2,2.5 & 98.76\text{–}99.20 & 95.67\text{–}97.24 & 10,10,9 \\\\ \hline s_{\max} & 2,3,4,5 & 97.56\text{–}98.04 & 96.68\text{–}97.22 & 10,10,10,10 \\\\ \end{array}

Under the 95%95\% MACs budget, all configurations achieve 1010/1010 feasible clients, with the average accuracy ranging from 97.46%97.46\% to 99.48%99.48\% . Under the 90%90\% MACs budget, the average accuracy ranges from 95.67%95.67\% to 98.70%98.70\%, and all smax⁡s_{\max} settings maintain 10/10 feasible clients.

Reviewer_7729OpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MDMP, a metadata-guided model modularization framework for resource-heterogeneous federated learning. The method uses conditional gradients to identify relationships between client data characteristics and model structures, groups structural units into functional modules, and assigns suitable modules to clients according to their data and computational budgets.

Experiments on MNIST, CelebA, and CIFAR-10 show that MDMP can reduce client-side computation while maintaining relatively small accuracy degradation, particularly at the 95% computation budget.

Critical strengths
  1. Considering both client data characteristics and computational resources when assigning model components is a meaningful direction for heterogeneous FL.

  2. I appreciate that the authors report actual MACs and worst-client accuracy drops, rather than relying only on average accuracy or parameter counts.

Critical weaknesses
  1. Appendix E describes a two-stage calibration procedure involving conditional gradients, Hessians, and additional client-server communication. Given that the reported computation savings are only around 6–9%, it is unclear whether these savings can offset the cost of constructing and calibrating the modules.

  2. In Table 1, MDMP performs worse than L1 pruning on MNIST and CIFAR-10 at nearly identical MACs. Moreover, the response-concentration analysis in Appendix D shows no clear improvement on CIFAR-10 after calibration. These observations raise questions about whether the proposed disentanglement mechanism consistently improves module selection.

Questions for authors
  1. Does the calibration procedure in Appendix E require computing and transmitting full Hessian matrices, or are Hessian-vector products used in practice? Could the authors clarify the actual computational and communication overhead?

  2. Is the calibration procedure performed only once, and approximately how many subsequent training rounds would be needed to offset its overhead?

  3. Appendix D reports that calibration does not improve response concentration on CIFAR-10. How do the authors explain this result, and is there existing ablation evidence that calibration improves the final module allocation?

  4. Eq. (4) uses gradient norms to identify the relationship between structural units and data characteristics. However, a large gradient may indicate a poorly fitted condition rather than a functionally important module. Conversely, well-trained modules may have small gradients despite being important for prediction. The authors should clarify how gradient sensitivity translates into functional importance and whether the two are empirically consistent.

  5. Eqs. (2)–(3) encourage structural units to concentrate their responses on fewer conditions. However, different data characteristics may naturally depend on shared network components. It is unclear whether enforcing response separation always benefits modularization or may remove useful shared information. This concern is particularly relevant because Appendix D shows no clear improvement in response concentration on CIFAR-10.

  6. Problem P2 selects modules according to metadata coverage and computational cost, but does not explicitly optimize the prediction loss of the resulting submodel. Covering all major data characteristics does not necessarily guarantee good performance, especially when modules have strong functional dependencies. The authors should explain why the set-cover formulation is sufficient and how interactions between selected modules are handled.

3 comments
AuthorsOpenReview ↗

Response (Part 1/3)

Thank you to the reviewers for your valuable feedback on our paper. We appreciate the time you spent reviewing our work and your insightful comments. This rebuttal will address the main points you raised. We're eager to clarify these points and improve our paper based on your suggestions.

Weakness 1 & Question 1: Does the calibration procedure in Appendix E require computing and transmitting full Hessian matrices, or are Hessian-vector products used in practice? Could the authors clarify the actual computational and communication overhead?

R: The implementation to obtain Hessian-vector products (HVPs) through automatic differentiation, rather than constructing or transmitting a full Hessian matrix. Clients obtain HVPs by differentiating the inner product of the gradient and a given direction vector, and upload one PP-dimensional result vector. The HVP result therefore requires O(P)O(P) storage and communication, rather than the O(P2)O(P^2) required for a full Hessian.

Computation and GPU memory costs: The evaluations of calibration, conditional-gradient calculation, and statistics-set response use the same RTX A6000, FP32, micro-batch size 32, and CUDA-synchronized timing. In Table 1, one calibration parameter update refers to as the model update time after the two-stage strategy, while HVP time includes the time of that update. Metadata processing includes reading metadata for MNIST and CelebA, and encoding CIFAR-10 training images with frozen CLIP. GPU memory denotes the peak allocated memory. Table 11 reports the detailed costs.

Table 1. Computation and memory costs. Cost itemMNISTCelebACIFAR-10Total calibration steps533Metadata reading / image encoding (s)0.3090.77025.019One calibration parameter update (s)606.494127.351223.81HVP time(s)478.393367.491018.81Conditional gradients on Dcal (s)88.80638.79168.07Peak calibration GPU memory (GiB)1.7423.8441.999Responses on Dstat (s)86.41635.4356.82Module-partition search (s)292.5216.94225.70Scoring all candidates (s)555.52770.73375.96Final selection (ms)28.3016.3534.26\begin{array}{lrrr}\hline \text{Cost item} & \text{MNIST} & \text{CelebA} & \text{CIFAR-10} \\\\ \hline \text{Total calibration steps} & 5 & 3 & 3 \\\\ \text{Metadata reading / image encoding (s)} & 0.309 & 0.770 & 25.019 \\\\ \text{One calibration parameter update (s)} & 606.49 & 4127.35 & 1223.81 \\\\ \quad\text{HVP time(s)} & 478.39 & 3367.49 & 1018.81 \\\\ \text{Conditional gradients on }D_{\mathrm{cal}}\text{ (s)} & 88.80 & 638.79 & 168.07 \\\\ \text{Peak calibration GPU memory (GiB)} & 1.742 & 3.844 & 1.999 \\\\ \text{Responses on }D_{\mathrm{stat}}\text{ (s)} & 86.41 & 635.43 & 56.82 \\\\ \text{Module-partition search (s)} & 292.52 & 16.94 & 225.70 \\\\ \text{Scoring all candidates (s)} & 555.52 & 770.73 & 375.96 \\\\ \text{Final selection (ms)} & 28.30 & 16.35 & 34.26 \\\\ \hline \end{array}

Communication cost: The total communication payload of a client for one calibration parameter update is denoted as

Bupdate=4CP⏟Gradient upload+4CP⏟Direction download+4P⏟Accumulated HVP upload+16C⏟Counts / losses upload=8CP+4P+16Cbytes.B_{\mathrm{update}} = \underbrace{4CP}_{\text{Gradient upload}} + \underbrace{4CP}_{\text{Direction download}} + \underbrace{4P}_{\text{Accumulated HVP upload}} + \underbrace{16C}_{\text{Counts / losses upload}} = 8CP + 4P + 16C\quad\text{bytes}.

Here, CC is the number of conditions features in each dataset and PP is the dimension of model parameter. The values of CC are 2828, 8080, and 2626 for MNIST, CelebA, and CIFAR-10, respectively. Table 22 reports the communication cost of each client in one calibration parameter update. The main communication costs come from conditional gradients and the vectors aq,jℓ,(t)\boldsymbol{a}_{q,j}^{\ell,(t)} sent from the server to clients==direction vectors rather than a full Hessian.

Table 2. Calibration communication per client per step. Payload itemMNISTCelebACIFAR-10Gradient upload (GiB)1.1653.3311.082Download of aq,jℓ,(t) (GiB)1.1653.3311.082HVP upload (MiB)42.62142.63942.625Counts / losses upload (bytes)4481280416Total payload (GiB)2.3726.7042.206\begin{array}{lrrr}\hline \text{Payload item} & \text{MNIST} & \text{CelebA} & \text{CIFAR-10} \\\\ \hline \text{Gradient upload (GiB)} & 1.165 & 3.331 & 1.082 \\\\ \text{Download of }\boldsymbol{a}_{q,j}^{\ell,(t)}\text{ (GiB)} & 1.165 & 3.331 & 1.082 \\\\ \text{HVP upload (MiB)} & 42.621 & 42.639 & 42.625 \\\\ \text{Counts / losses upload (bytes)} & 448 & 1280 & 416 \\\\ \text{Total payload (GiB)} & 2.372 & 6.704 & 2.206 \\\\ \hline \end{array}

Weakness 1 & Question 2: It is unclear whether these savings can offset the cost of constructing and calibrating the modules. Is the calibration procedure performed only once, and approximately how many subsequent training rounds would be needed to offset its overhead?

AuthorsOpenReview ↗

Response (Part 2/3)

R: Thank you for the question. We are very sorry for confusing you on the whole procedurces of our proposed method. Actually, MDMP allocates modules based on client data characteristics and resource budgets, with the goal of enabling resource-constrained clients to participate. For a given global model, calibration is performed once before formal deployment in practice, rather than repeating in every subsequent training round. The MDMP procedure includes three components in the overview diagram in Appendix JJ: (a) functional module construction, which starts from a pretrained federated global model and performs response calibration, conditional-response measurement, and module partitioning in advance; (b) Client sub-module determination, which selects modules from the existing library based on client's metadata distribution and MACs budget; (c) Participation by resource-constrained clients, which assigns submodels that satisfies MACs budget to clients, enabling local model training. Note that **in the above three steps, step (a) and (b) should be "trained" on some clients in advance before later deployment in FL to allow resource-constrained clients to participant.

Evidence of participation for resource-constrained clients: In Table 33, we added 55 resource-constrained clients to each of the three datasets and conducted each test 3 times, resulting in a total of 45 tests.The maximal GPU memory limits are 220220, 216216, and 238238 MiB respectively. The peak training memory usages of the MDMP sub-models are 182.23182.23, 196.22196.22, and 214.18214.18 MiB respectively, all below these limits.Under the same training conditions, the complete global model failed to complete the training 4545 times due to out-of-memory (OOM). However, the MDMP sub-model completed local training and testing 4545 times, indicating that MDMP enables these resource-constrained clients that were previously unable to train the global model to participate.

Table 3. Resource-constrained participation. ItemMNISTCelebACIFAR-10Batch size163216Maximal GPU memory (MiB)220216238Peak training memory (MiB)182.23196.22214.18Full-model OOM15/1515/1515/15MDMP completed15/1515/1515/15Mean accuracy drop (pp)3.1420.1863.750\begin{array}{lrrr}\text{Item} & \text{MNIST} & \text{CelebA} & \text{CIFAR-10} \\\\ \hline \text{Batch size} & 16 & 32 & 16 \\\\ \hline \text{Maximal GPU memory (MiB)} & 220 & 216 & 238 \\\\ \hline \text{Peak training memory (MiB)} & 182.23 & 196.22 & 214.18 \\\\ \hline \text{Full-model OOM} & 15/15 & 15/15 & 15/15 \\\\ \hline \mathbf{\text{MDMP completed}} & 15/15 & 15/15 & 15/15 \\\\ \hline \text{Mean accuracy drop (pp)} & 3.142 & 0.186 & 3.750\end{array}

Weakness 2 & Question 3: Appendix D reports that calibration does not improve response concentration on CIFAR-10. How do the authors explain this result, and is there existing ablation evidence that calibration improves the final module allocation?

R: Thank you for asking about CIFAR-10 calibration and its role in module allocation.

Response concentration on CIFAR-10: On CIFAR-10, within-factor response entropy decreases from 0.8586860.858686 to 0.8569220.856922, and the separation loss remains zero before and after calibration, indicating that the current model already satisfies the separation threshold, as demonstrated in Table 44, where calibration uses Dkcal\mathcal D_k^{\mathrm{cal}}, followed by response evaluation on Dkstat\mathcal D_k^{\mathrm{stat}}. HH is the mean normalized within-factor response entropy on Dkstat\mathcal D_k^{\mathrm{stat}}, and Lsep\mathcal L_{\mathrm{sep}} is the separation loss on Dkcal\mathcal D_k^{\mathrm{cal}}. Subscripts 0/10/1 denote before and after calibration. However, on MNIST and CelebA, response entropy decreases, and the separation term contributes to the calibration objective.

Table 4. Response concentrations. DatasetH0H1Lsep,0Lsep,1MNIST0.8823300.8579786.110!×!10−44.712!×!10−4CelebA0.7914880.7889137.245!×!10−47.513!×!10−4CIFAR-100.8586860.85692200\begin{array}{lrrrr}\text{Dataset} & H_0 & H_1 & \mathcal{L}_{\mathrm{sep},0} & \mathcal{L}_{\mathrm{sep},1} \\\\ \hline \text{MNIST} & 0.882330 & 0.857978 & 6.110\\!\times\\!10^{-4} & 4.712\\!\times\\!10^{-4} \\\\ \hline \text{CelebA} & 0.791488 & 0.788913 & 7.245\\!\times\\!10^{-4} & 7.513\\!\times\\!10^{-4} \\\\ \hline \text{CIFAR-10} & 0.858686 & 0.856922 & 0 & 0\end{array}

Calibration ablation: We compare no calibration (NC), task loss plus the proximal term only (TLP), and full calibration (FC) in terms of MACs computation volume and the number of feasible clients.Here, TLP and FC use the same update steps and learning rates. Only FC incorporates the structure calibration term to distinguish the effects of structure calibration from simple task training.The first three columns in Table 55 show the computational costs of MACs for the three groups of results; Feas lists the number of feasible clients in the order of the three groups.

Table 5. Calibration ablation. DatasetNCTLPFCFeasMNIST94.73794.73885.1951/5/10CelebA89.17389.18688.6138/8/10CIFAR-1086.26278.70885.1349/10/10\begin{array}{lrrrr}\text{Dataset} & \text{NC} & \text{TLP} & \text{FC} & \text{Feas} \\\\ \hline \text{MNIST} & 94.737 & 94.738 & 85.195 & 1/5/10 \\\\ \hline \text{CelebA} & 89.173 & 89.186 & 88.613 & 8/8/10 \\\\ \hline \text{CIFAR-10} & 86.262 & 78.708 & 85.134 & 9/10/10\end{array}

AuthorsOpenReview ↗

Response (Part 3/3)

Compared to TLP, FC can achieve full 10-client functionality with lower computational cost on MNIST and CelebA, indicating that structure calibration is conducive to module allocation with lower computational requirements.

Question 4: Eq. (4) uses gradient norms to identify the relationship between structural units and data characteristics. However, a large gradient may indicate a poorly fitted condition rather than a functionally important module. Conversely, well-trained modules may have small gradients despite being important for prediction. The authors should clarify how gradient sensitivity translates into functional importance and whether the two are empirically consistent.

R: Thank you for this insightful comment. We agree that large gradients can reflect insufficient fitting of a condition, while well-trained structures important for prediction can have small gradients. In our paper, the conditional gradient is applied to evaluate the sensitivity of client's data on different parts of model. Our idea comes from that if some parts of a model's conditional gradients are sensitivity to the dataset, the corresponding model parameters can be changed and updated based on gradient descent method. However, for the zero or small values of conditional gradients, which means insensitivity to a part of the model, the corresponding model parameters will merely unchanged during model training. To this end, those model paramters are not necessary to transmit to this client for local model training, as those paramters are not changed.

Question 5: Eqs. (2)–(3) encourage structural units to concentrate their responses on fewer conditions. However, different data characteristics may naturally depend on shared network components. It is unclear whether enforcing response separation always benefits modularization or may remove useful shared information. This concern is particularly relevant because Appendix D shows no clear improvement in response concentration on CIFAR-10.

R: Thank the reviewers for pointing out that different data features may rely on shared components.We will explain from three aspects: constraint design, sharing mechanism, and ablation evidence.

Equations (2)-(3) preserve room for sharing: Eq (2) calculates the normalized response entropy within each factor. The same structural unit can respond to multiple conditions from different factors.The same structural unit can respond to multiple conditions from different factors. For example, it can respond to both a class and related visual attributes. Equation (3) acts on the conditions within the same factor where the similarity exceeds the threshold, retaining the response similarity within the threshold range and the functional sharing across factors.The task loss and the proximal term are simultaneously constrained for calibration, enabling the response structure adjustment to take into account both the prediction performance and parameter offset.

Module partitioning supports functional sharing: Although the structural units are mutually exclusive and grouped, the condition sets of different modules can still overlap, and the same module can serve multiple conditions and clients. The shared units and the selected modules together form the client sub-model.

Ablation evidence: As shown in Question 33 and Table 44, the separation loss of CIFAR-10 remained zero from before calibration, and this penalty remained zero in this run; the intra-factor response entropy changed from 0.8586860.858686 to 0.8569220.856922, which is consistent with the smaller response change in Appendix D.This indicates that this operation did not achieve modularity through continuous enhancement of separation penalties. Table 55 further provides direct evidence of the calibration on the final allocation: FC can achieve all 10 client nodes with the same budget as TLP on MNIST and CelebA, supporting the contribution of calibration to the modular allocation.

Question 6: Problem P2 selects modules according to metadata coverage and computational cost, but does not explicitly optimize the prediction loss of the resulting submodel. Covering all major data characteristics does not necessarily guarantee good performance, especially when modules have strong functional dependencies. The authors should explain why the set-cover formulation is sufficient and how interactions between selected modules are handled?

R: Thank you for asking us to clarify this implementation details. Through extensive simulation observations, we have found that the sub-models composed of the selected functional modules of the client can cover the main conditional characteristics of the client's metadata, and can maintain that the accuracy of the sub-model relative to the calibrated global convergence model is not significantly reduced. Therefore, we adopt set coverage.

Reviewer_mS1vOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies how to assign smaller models to federated clients with different data characteristics and computational budgets. Starting from a trained global model, MDMP calibrates gradient responses to metadata conditions and groups model components with similar responses into functional modules. It then selects modules that cover each client’s main data characteristics while satisfying its computational budget. Experiments on MNIST, CelebA, and CIFAR-10 report modest computational savings relative to the full model, with generally limited accuracy loss.

Critical strengths
  1. Relevant problem formulation: Considering both local data characteristics and computational capacity provides a useful perspective on assigning submodels to heterogeneous clients. The explicit connection between metadata and model components gives the proposed allocation strategy an understandable rationale.
Critical weaknesses
  1. Incomplete related work and unclear added value: Section 2 omits relevant methods such as FIARSE (NeurIPS 2024), FedSelect (CVPR 2024), and SubFLOT (CVPR 2026), which address importance-based submodel extraction, personalized parameter selection, and data-adaptive federated pruning. These works should be cited and discussed to clarify what explicit metadata-conditioned modularization adds beyond existing selection mechanisms. The matched-MAC random control in Appendix I.8 also fails to demonstrate a consistent advantage for response-based grouping. More targeted component ablations are needed to establish the benefits of calibration, metadata, and the proposed grouping procedure.
  2. Unclear feasibility for resource-constrained clients: The proposed MDMP requires local full-model computations during calibration and response estimation before client subnetworks are assigned (Sections 4.1--4.2 and Appendix E). The procedure therefore leaves unresolved how clients unable to execute the full model can participate, even if a pretrained checkpoint is available. The MAC-based resource constraints do not establish memory feasibility, and the reported subnetwork savings do not account for the full preparation cost. The authors should clarify the minimum client capabilities assumed and demonstrate how clients with insufficient memory for these full-model computations can obtain their subnetworks.
  3. Limited comparative evidence and uncertain practical savings: In Table 1, MDMP trails L1 pruning on MNIST and CIFAR-10, while Tables 2 and 5 show that HeteroFL achieves higher accuracy with fewer measured MACs on both datasets. Although the MDMP seed repetitions are useful, the baseline comparisons remain single-seed, and robustness to noisy or missing metadata is untested. The reported 6.38–9.24% MAC reductions describe the selected subnetworks without establishing the total cost of metadata extraction, calibration, and module selection. Since MDMP also starts from a trained global model, its net training savings and practical benefit for resource-constrained clients remain unclear.
Questions for authors
  1. Please cite and discuss FIARSE, FedSelect, and SubFLOT, explaining the differences in assumptions, selection mechanisms, and computational requirements. Can you also compare them with the closest methods under compatible protocols, along with matched-MAC ablations that isolate calibration, metadata-based allocation, and response-based grouping?
  2. How can a client that cannot execute the full model perform the calibration and conditional-gradient estimation required before subnetwork allocation? Please clarify whether you support such clients, identify where these computations would take place, and report peak client memory and preparation cost, including the second-order calculations in Appendix E.
  3. Do the comparative conclusions hold across matched random seeds, equal measured computation, and noisy or incomplete metadata? Please clarify who performs the initial full-model training and report the computation, communication, and runtime overhead of preparing the client subnetworks. Under the intended deployment conditions, how much subsequent training is needed to recover that preparation cost?
8 comments
AuthorsOpenReview ↗

Response (Part 1/5)

Thank you to the reviewers for your valuable feedback on our paper. We appreciate the time you spent reviewing our work and your insightful comments. This rebuttal will address the main points you raised.

Weakness 1 & Question 1:** Please cite and discuss FIARSE, FedSelect, and SubFLOT, explaining the differences in assumptions, selection mechanisms, and computational requirements. Can you also compare them with the closest methods under compatible protocols, along with matched-MAC ablations that isolate calibration, metadata-based allocation, and response-based grouping?

R: Thank you for suggesting that we add relevant work, method comparisons under compatible protocols, and component ablation experiments. Table 11 summarizes the differences among FIARSE [1], FedSelect [2], SubFLOT [3] and MDMP in terms of the required conditions, selection mechanisms, and computational requirements.

Table 1. Method comparison. CriterionFIARSEFedSelectSubFLOTMDMPAssumptionParametercapacityLocal parameterretentionHistorical modelsavailableGlobal model+ metadataMetricParametermagnitudeLocal updatesHistorical modelmatchingConditionalresponsesUnitParameterParameterNeuron/channelModuleModule library×××✓ConstraintParametercapacityPersonalizationratioPruning ratioMACs budgetComputationThresholding,mask updatesUpdate comparison,parameter selectionOptimal transport,regularizationCalibration,response matrix\begin{array}{lcccc}\text{Criterion} & \text{FIARSE} & \text{FedSelect} & \text{SubFLOT} & \text{MDMP} \\\\ \hline \text{Assumption} & \begin{array}{c}\text{Parameter}\\\\\text{capacity}\end{array} & \begin{array}{c}\text{Local parameter}\\\\\text{retention}\end{array} & \begin{array}{c}\text{Historical models}\\\\\text{available}\end{array} & \begin{array}{c}\text{Global model}\\\\\text{+ metadata}\end{array} \\\\ \hline \text{Metric} & \begin{array}{c}\text{Parameter}\\\\\text{magnitude}\end{array} & \text{Local updates} & \begin{array}{c}\text{Historical model}\\\\\text{matching}\end{array} & \begin{array}{c}\text{Conditional}\\\\\text{responses}\end{array} \\\\ \hline \text{Unit} & \text{Parameter} & \text{Parameter} & \text{Neuron/channel} & \text{Module} \\\\ \hline \text{Module library} & \times & \times & \times & \checkmark \\\\ \hline \text{Constraint} & \begin{array}{c}\text{Parameter}\\\\\text{capacity}\end{array} & \begin{array}{c}\text{Personalization}\\\\\text{ratio}\end{array} & \text{Pruning ratio} & \text{MACs budget} \\\\ \hline \text{Computation} & \begin{array}{c}\text{Thresholding,}\\\\\text{mask updates}\end{array} & \begin{array}{c}\text{Update comparison,}\\\\\text{parameter selection}\end{array} & \begin{array}{c}\text{Optimal transport,}\\\\\text{regularization}\end{array} & \begin{array}{c}\text{Calibration,}\\\\\text{response matrix}\end{array} \\\\ \hline \end{array}

FIARSE and FedSelect make selections at the parameter level, while SubFLOT generates structured sub-models for the client side; MDMP builds module libraries based on conditional responses, and combines them on demand within the MACs budget according to metadata, enabling the same module library to be adapted for different clients.

[1] Wu et al. FIARSE: Model-Heterogeneous Federated Learning via Importance-Aware Submodel Extraction. NeurIPS, 2024.

[2] Tamirisa et al. FedSelect: Personalized Federated Learning with Customized Selection of Parameters for Fine-Tuning. CVPR, 2024.

[3] Jiang et al. Submodel Extraction for Efficient and Personalized Federated Learning via Optimal Transport. CVPR, 2026.

Comparison Method and Settings: We measure the computational load of the client sub-models in terms of MACs and compare their average test accuracy. Since the implementation of FIARSE involves setting the mask to zero as a parameter, it still performs dense operations in the original shape. Therefore, its sparsity rate cannot be directly equivalent to the reduction in MACs. FedSelect divides the shared and personalized parameters, and both parts still jointly participate in the prediction. SubFLOT shrinks the model through neuron and channel pruning, hence we choose it for comparison.

The values in Table 22 represent the average test accuracy (%) and computational cost (%) of 10 clients under three datasets, with MACs Budget indicating the upper limit of computational cost. Since the computational cost after SubFLOT pruning cannot be made consistent with that of MDMP, we trained models with slightly lower (L) and slightly higher (U) computational costs respectively to examine whether the difference in accuracy stems from the difference in computational cost.

Table 2. Comparison with SubFLOT. \begin{array}{lcrrr}\text{Dataset} & \text{MACs Budget} & \text{MDMP} & \text{SubFLOT-L} & \text{SubFLOT-U} \\\\ \hline \text{MNIST} & 95\\% & 99.535/93.623 & 99.014/93.410 & 99.142/93.643 \\\\ \hline \text{MNIST} & 90\\% & 98.926/85.924 & 98.041/85.516 & 97.752/86.118 \\\\ \hline \text{CelebA} & 95\\% & 92.377/88.726 & 90.050/88.523 & 89.800/90.004 \\\\ \hline \text{CelebA} & 90\\% & 92.334/87.035 & 89.776/86.805 & 89.330/88.272 \\\\ \hline \text{CIFAR-10} & 95\\% & 87.813/92.527 & 54.413/92.016 & 56.413/93.398 \\\\ \hline \text{CIFAR-10} & 90\\% & 87.275/89.843 & 43.888/89.366 & 49.763/90.711\end{array}

AuthorsOpenReview ↗

Response (Part 2/5)

The results show that MDMP achieved a higher average test accuracy with a similar computational load as SubFLOT.

Calibration ablation:** We compare no calibration (NC), task loss plus the proximal term only (TLP), and full calibration (FC) in terms of MACs computation volume and the number of feasible clients.Here, TLP and FC use the same update steps and learning rates. Only FC incorporates the structure calibration term to distinguish the effects of structure calibration from simple task training. The first three columns in Table 33 show the computational costs of MACs for the three groups of results; Feas lists the number of feasible clients in the order of the three groups.

Table 3. Calibration ablation. DatasetNCTLPFCFeasMNIST94.73794.73885.1951/5/10CelebA89.17389.18688.6138/8/10CIFAR-1086.26278.70885.1349/10/10\begin{array}{lrrrr}\text{Dataset} & \text{NC} & \text{TLP} & \text{FC} & \text{Feas} \\\\ \hline \text{MNIST} & 94.737 & 94.738 & 85.195 & 1/5/10 \\\\ \hline \text{CelebA} & 89.173 & 89.186 & 88.613 & 8/8/10 \\\\ \hline \text{CIFAR-10} & 86.262 & 78.708 & 85.134 & 9/10/10\end{array}

Compared to TLP, FC can achieve full 10-client functionality with lower computational cost on MNIST and CelebA, indicating that structure calibration is conducive to module allocation with lower computational requirements.

Ablation of response-based grouping and metadata: We added two types of benchmarks:

(1) Random grouping: etain the calibration weights and replace the response groups with independent random partitions that satisfy the same parameter and MAC capacity constraints.

(2) No-metadata pipeline: starting from the same original global model, remove the modules for conditional calibration, response matrix calculation, and selection of client metadata, and complete the distribution using random partitioning and ordinary task loss.

Table 44 reports the number of feasible clients under each MACs budget, with 10 clients in each group. Both of these benchmark schemes were randomly divided into 3 groups. The three numbers in the table represent the number of feasible clients for each time, with MNIST, CelebA, and CIFAR-10 using M=8,4,20M=8, 4, 20, respectively.

Table 4. Grouping and metadata ablations. \begin{array}{lcrcc}\text{Dataset} & \text{MACs Budget} & \text{MDMP} & \text{Random grouping} & \text{No-metadata pipeline} \\\\ \hline \text{MNIST} & 95\\% & 10/10 & 0,0,1 & 0,0,1 \\\\ \hline \text{MNIST} & 90\\% & 9/10 & 0,0,1 & 0,0,1 \\\\ \hline \text{CelebA} & 95\\% & 10/10 & 0,0,0 & 0,0,0 \\\\ \hline \text{CelebA} & 90\\% & 10/10 & 0,0,0 & 0,0,0 \\\\ \hline \text{CIFAR-10} & 95\\% & 10/10 & 10,10,10 & 10,10,10 \\\\ \hline \text{CIFAR-10} & 90\\% & 10/10 & 10,10,10 & 10,10,10\end{array}

On MNIST and CelebA datasets, the number of feasible clients for MDMP is higher than that of the two baseline schemes, supporting the contribution of response grouping and metadata to the module allocation within the MAC budget. On CIFAR-10, all three groups are 1010/1010. Since it is already decoupled by itself, the two baseline schemes have almost no impact compared to MDMP.

Weakness 2 & Question 2 : Unclear feasibility for resource-constrained clients: The proposed MDMP requires local full-model computations during calibration and response estimation before client subnetworks are assigned (Sections 4.1--4.2 and Appendix E). The procedure therefore leaves unresolved how clients unable to execute the full model can participate, even if a pretrained checkpoint is available. The MAC-based resource constraints do not establish memory feasibility, and the reported subnetwork savings do not account for the full preparation cost. The authors should clarify the minimum client capabilities assumed and demonstrate how clients with insufficient memory for these full-model computations can obtain their subnetworks.

R:** Thank you for the question. We are very sorry for confusing you on the whole procedurces of our proposed method. Actually, MDMP allocates modules based on client data characteristics and resource budgets, with the goal of enabling resource-constrained clients to participate. For a given global model, calibration is performed once before formal deployment in practice, rather than repeating in every subsequent training round. The MDMP procedure includes three components in the overview diagram in Appendix J: (a) functional module construction, which starts from a pretrained federated global model and performs response calibration, conditional-response measurement, and module partitioning in advance; (b) Client sub-module determination, which selects modules from the existing library based on client's metadata distribution and MACs budget; (c) Participation by resource-constrained clients, which assigns submodels that satisfies MACs budget to clients, enabling local model training.

AuthorsOpenReview ↗

Response (Part 3/5)

Note that in the above three steps, step (a) and (b) should be "trained" on some clients in advance before later deployment in FL to allow resource-constrained clients to participant. Therefore, resource-constrained clients do not perform calibration and condition gradient estimation. Next, we will report the expenses for the calibration process.

Computation and GPU memory costs: The evaluations of calibration, conditional-gradient calculation, and statistics-set response use the same RTX A6000, FP32, micro-batch size 32, and CUDA-synchronized timing. In Table 55, one calibration parameter update refers to as the model update time after the two-stage strategy, while HVP time includes the time of that update. Metadata processing includes reading metadata for MNIST and CelebA, and encoding CIFAR-10 training images with frozen CLIP. GPU memory denotes the peak allocated memory. Table 55 reports the detailed costs.

**Table 5. Computation and memory costs. Cost itemMNISTCelebACIFAR-10Total calibration steps533Metadata reading / image encoding (s)0.3090.77025.019One calibration parameter update (s)606.494127.351223.81HVP time(s)478.393367.491018.81Conditional gradients on Dcal (s)88.80638.79168.07Peak calibration GPU memory (GiB)1.7423.8441.999Responses on Dstat (s)86.41635.4356.82Module-partition search (s)292.5216.94225.70Scoring all candidates (s)555.52770.73375.96Final selection (ms)28.3016.3534.26\begin{array}{lrrr}\hline \text{Cost item} & \text{MNIST} & \text{CelebA} & \text{CIFAR-10} \\\\ \hline \text{Total calibration steps} & 5 & 3 & 3 \\\\ \text{Metadata reading / image encoding (s)} & 0.309 & 0.770 & 25.019 \\\\ \text{One calibration parameter update (s)} & 606.49 & 4127.35 & 1223.81 \\\\ \quad\text{HVP time(s)} & 478.39 & 3367.49 & 1018.81 \\\\ \text{Conditional gradients on }D_{\mathrm{cal}}\text{ (s)} & 88.80 & 638.79 & 168.07 \\\\ \text{Peak calibration GPU memory (GiB)} & 1.742 & 3.844 & 1.999 \\\\ \text{Responses on }D_{\mathrm{stat}}\text{ (s)} & 86.41 & 635.43 & 56.82 \\\\ \text{Module-partition search (s)} & 292.52 & 16.94 & 225.70 \\\\ \text{Scoring all candidates (s)} & 555.52 & 770.73 & 375.96 \\\\ \text{Final selection (ms)} & 28.30 & 16.35 & 34.26 \\\\ \hline \end{array}

Communication cost: The total communication payload of a client for one calibration parameter update is denoted as Bupdate=4CP⏟Gradient upload+4CP⏟Direction download+4P⏟Accumulated HVP upload+16C⏟Counts / losses upload=8CP+4P+16Cbytes.B_{\mathrm{update}} = \underbrace{4CP}_{\text{Gradient upload}} + \underbrace{4CP}_{\text{Direction download}} + \underbrace{4P}_{\text{Accumulated HVP upload}} + \underbrace{16C}_{\text{Counts / losses upload}} = 8CP + 4P + 16C\quad\text{bytes}.

Here, CC is the number of conditions features in each dataset and PP is the dimension of model parameter. The values of CC are 2828, 8080, and 2626 for MNIST, CelebA, and CIFAR-10, respectively. Table 66 reports the communication cost of each client in one calibration parameter update. The main communication costs come from conditional gradients and the vectors aq,jℓ,(t)\boldsymbol{a}_{q,j}^{\ell,(t)} sent from the server to clients==direction vectors rather than a full Hessian.

Table 6. Calibration communication per client per step. Payload itemMNISTCelebACIFAR-10Gradient upload (GiB)1.1653.3311.082Download of aq,jℓ,(t) (GiB)1.1653.3311.082HVP upload (MiB)42.62142.63942.625Counts / losses upload (bytes)4481280416Total payload (GiB)2.3726.7042.206\begin{array}{lrrr}\hline \text{Payload item} & \text{MNIST} & \text{CelebA} & \text{CIFAR-10} \\\\ \hline \text{Gradient upload (GiB)} & 1.165 & 3.331 & 1.082 \\\\ \text{Download of }\boldsymbol{a}_{q,j}^{\ell,(t)}\text{ (GiB)} & 1.165 & 3.331 & 1.082 \\\\ \text{HVP upload (MiB)} & 42.621 & 42.639 & 42.625 \\\\ \text{Counts / losses upload (bytes)} & 448 & 1280 & 416 \\\\ \text{Total payload (GiB)} & 2.372 & 6.704 & 2.206 \\\\ \hline \end{array}

Evidence of participation for resource-constrained clients: In Table 77, we add 55 resource-constrained clients to each of the three datasets and conduct each test 3 times, resulting in a total of 45 tests. The maximal GPU memory limits are 220220, 216216, and 238238 MiB respectively. The peak training memory usages of the MDMP sub-models are 182.23182.23, 196.22196.22, and 214.18214.18 MiB respectively, all below these limits.Under the same training conditions, the complete global model failed to complete the training 4545 times due to out-of-memory (OOM). However, the MDMP sub-model completed local training and testing 4545 times, indicating that MDMP enables these resource-constrained clients that were previously unable to train the global model to participate.

Table 7. Resource-constrained participation. ItemMNISTCelebACIFAR-10Batch size163216Maximal GPU memory (MiB)220216238Peak training memory (MiB)182.23196.22214.18Full-model OOM15/1515/1515/15MDMP completed15/1515/1515/15Mean accuracy drop (pp)3.1420.1863.750\begin{array}{lrrr}\text{Item} & \text{MNIST} & \text{CelebA} & \text{CIFAR-10} \\\\ \hline \text{Batch size} & 16 & 32 & 16 \\\\ \hline \text{Maximal GPU memory (MiB)} & 220 & 216 & 238 \\\\ \hline \text{Peak training memory (MiB)} & 182.23 & 196.22 & 214.18 \\\\ \hline \text{Full-model OOM} & 15/15 & 15/15 & 15/15 \\\\ \hline \mathbf{\text{MDMP completed}} & 15/15 & 15/15 & 15/15 \\\\ \hline \text{Mean accuracy drop (pp)} & 3.142 & 0.186 & 3.750\end{array}

Comments on the paper

3 comments
AuthorsOpenReview ↗

Global response (part 1/3)

We thank reviewers 8sQp, 7299, and m1sv for their careful reading and specific suggestions. We address the concerns shared by 7299 and m1sv regarding calibration and grouping, baseline comparisons, preparation costs, and participation by resource-constrained clients. We present the additional experiments and implementation procedurces for the reviewers to examine. Other specific questions are addressed in the individual responses.

1. Concerns on the contributions of calibration and response-based grouping to the final module allocation

Shared concern: Reviewer 7299, Weakness 2 and Question 3, concerning the calibration result on CIFAR-10 and the role of calibration; Reviewer m1sv, Weakness 1 and Question 1, requesting ablations of calibration, response-based grouping, and the metadata-driven pipeline.

To address the ablation studies, we added three settings: no calibration (NC), task loss plus the proximal term only (TLP), and full calibration (FC). The latter two use the same number of updates and learning rate. Full calibration reduces the uniform MACs budget required for feasible allocation to all clients from 94.738%94.738\% to 85.195%85.195\% on MNIST and from 89.186%89.186\% to 88.613%88.613\% on CelebA. Under the above MACs budget, the numbers of feasible clients increase from 5/105/10 and 8/108/10 to 10/1010/10, respectively. The corresponding sub-models achieve averaged test accuracies of 97.661%97.661\% and 92.441%92.441\%. The three tables below respectively present the comparisons of response entropy and separation loss before and after calibration, as well as the final distribution and random module partitions. They correlate the effects of calibration and response grouping with the actual prediction results.

Response concentrations: On CIFAR-10, within-factor response entropy decreases from 0.8586860.858686 to 0.8569220.856922, and the separation loss remains zero before and after calibration, indicating that the current model already satisfies the separation threshold, as demonstrated in Table 11. In Table 11, calibration uses Dkcal\mathcal D_k^{\mathrm{cal}}, followed by response evaluation on Dkstat\mathcal D_k^{\mathrm{stat}}. HH is the mean normalized within-factor response entropy on Dkstat\mathcal D_k^{\mathrm{stat}}, and Lsep\mathcal L_{\mathrm{sep}} is the separation loss on Dkcal\mathcal D_k^{\mathrm{cal}}. Subscripts 0/10/1 denote before and after calibration. However, on MNIST and CelebA, response entropy decreases, and the separation term contributes to the calibration objective.

Table 1. Response concentrations.

DatasetH0H1Lsep,0Lsep,1MNIST0.8823300.8579786.110×10−44.712×10−4CelebA0.7914880.7889137.245×10−47.513×10−4CIFAR-100.8586860.85692200\begin{array}{lrrrr}\text{Dataset} & H_0 & H_1 & \mathcal{L}_{\mathrm{sep},0} & \mathcal{L}_{\mathrm{sep},1} \\\\ \hline \text{MNIST} & 0.882330 & 0.857978 & 6.110 \times 10^{-4} & 4.712 \times 10^{-4} \\\\ \hline \text{CelebA} & 0.791488 & 0.788913 & 7.245 \times 10^{-4} & 7.513 \times 10^{-4} \\\\ \hline \text{CIFAR-10} & 0.858686 & 0.856922 & 0 & 0\end{array}

Calibration ablation: We compare no calibration (NC), task loss plus the proximal term only (TLP), and full calibration (FC) in terms of MACs computation volume and the number of feasible clients.Here, TLP and FC use the same update steps and learning rates. Only FC incorporates the structure calibration term to distinguish the effects of structure calibration from simple task training.The first three columns in Table 22 show the computational costs of MACs for the three groups of results; Feas lists the number of feasible clients in the order of the three groups.

Table 2. Calibration ablation. DatasetNCTLPFCFeasMNIST94.73794.73885.1951/5/10CelebA89.17389.18688.6138/8/10CIFAR-1086.26278.70885.1349/10/10\begin{array}{lrrrr}\text{Dataset} & \text{NC} & \text{TLP} & \text{FC} & \text{Feas} \\\\ \hline \text{MNIST} & 94.737 & 94.738 & 85.195 & 1/5/10 \\\\ \hline \text{CelebA} & 89.173 & 89.186 & 88.613 & 8/8/10 \\\\ \hline \text{CIFAR-10} & 86.262 & 78.708 & 85.134 & 9/10/10\end{array}

Compared to TLP, FC can achieve full 10-client functionality with lower computational cost on MNIST and CelebA, indicating that structure calibration is conducive to module allocation with lower computational requirements.

Ablation of response-based grouping and metadata: We added two types of benchmarks:

(1) Random gruping: retain the calibration weights and replace the response groups with independent random partitions that satisfy the same parameter and MAC capacity constraints.

(2) No-metadata pipeline: starting from the same original global model, remove the modules for conditional calibration, response matrix calculation, and selection of client metadata, and complete the distribution using random partitioning and ordinary task loss.

Table 33 reports the number of feasible clients under each MACs budget, with 10 clients in each group.Both of these benchmark schemes were randomly divided into 3 groups. The three numbers in the table represent the number of feasible clients for each time, with MNIST, CelebA, and CIFAR-10 using M=8,4,20M=8, 4, 20, respectively.

AuthorsOpenReview ↗

Global response (part 2/3)

On MNIST and CelebA datasets, the number of feasible clients for MDMP is higher than that of the two baseline schemes, supporting the contribution of response grouping and metadata to the module allocation within the MAC budget. On CIFAR-10, all three groups are 1010/1010. Since it is already decoupled by itself, the two baseline schemes have almost no impact compared to MDMP.

2. Concerns on comparative evidence of L1, random seeds, and actual MACs

Shared concern: Reviewer 7299, Weakness 2, concerning the comparison with L1; Reviewer m1sv, Weakness 3 and Question 3, requesting stronger comparative evidence, random-seed repetitions, and actual computation measurements.

We add comparisons with three paired training seeds (0,1,20,1,2) at two budgets and report the MACs of the models. The maximum per-client MAC difference for the L1 control is below 0.0190.019 percentage points of full-model computation. At the two CelebA budgets and the 90%90\% CIFAR-10 budget, MDMP achieves mean accuracies across the three seeds of 92.511%92.511\%, 92.478%92.478\%, and 87.500%87.500\%, compared with 92.381%92.381\%, 92.402%92.402\%, and 87.367%87.367\% for L1. Table 44 reports all three datasets and both budgets. Accuracy is reported as the mean and sample standard deviation across three seeds, after equally averaging the 10 clients within each seed. MACs are percentages of full-model computation.

Table 4. Three-seed comparison with L1. \begin{array}{lcrrrr}\text{Dataset} & \text{Budget} & \text{MDMP Acc.} & \text{L1 Acc.} & \text{MDMP MACs} & \text{L1 MACs} \\\\ \hline \text{MNIST} & 95\\% & 98.805\pm0.346 & 99.612\pm0.062 & 93.495 & 93.488 \\\\ \hline \text{MNIST} & 90\\% & 97.580\pm0.530 & 99.511\pm0.046 & 87.896 & 87.883 \\\\ \hline \text{CelebA} & 95\\% & 92.511\pm0.083 & 92.381\pm0.209 & 90.756 & 90.747 \\\\ \hline \text{CelebA} & 90\\% & 92.478\pm0.050 & 92.402\pm0.220 & 86.958 & 86.947 \\\\ \hline \text{CIFAR-10} & 95\\% & 87.696\pm0.931 & 87.933\pm0.774 & 92.393 & 92.386 \\\\ \hline \text{CIFAR-10} & 90\\% & 87.500\pm0.513 & 87.367\pm0.803 & 89.207 & 89.203\end{array}

Under the two budget scenarios of MNIST and CelebA, MDMP achieved an average accuracy difference of 0.0240.024–0.3200.320 percentage points, while reducing parameters by 8.01%8.01\%–20.21%20.21\%.Under the two budget levels of CIFAR-10, the average accuracy was 0.3210.321 and 0.7960.796 percentage points lower respectively. However, the standard deviation of accuracy among the seeds was smaller, indicating a smaller fluctuation in training among the seeds.

3. Concerns on the process of our method and its costs.

Shared concern: Reviewer 7299, Weakness 1 and Questions 1 and 2, concerning he cost of the two-stage calibration process outlined in Appendix E; Reviewer m1sv, Weakness 2 and Question 2, concerning the issue where resource-constrained clients are unable to execute the complete model, and Weakness 3, concerning the issue of when the cost of the calibration process can be recovered.

Thank you for this comment. We should clarify that this paper aims to propose a model partition method in FL, and this method should be first "trained" by numbers of clients capable of training the full model. Then, in the later practical deployment, those resource-constrained clients only need to provide their metadata to the server, based on which the sub-model will be determined. Tables 66 and 77 report "training" computation and communication costs of our method, respectively, and Table 88 reports actual participation of resource-constrained clients.

Computation and GPU memory costs: The evaluations of calibration, conditional-gradient calculation, and statistics-set response use the same RTX A6000, FP32, micro-batch size 32, and CUDA-synchronized timing. In Table 1, one calibration parameter update refers to as the model update time after the two-stage strategy, while HVP time includes the time of that update. Metadata processing includes reading metadata for MNIST and CelebA, and encoding CIFAR-10 training images with frozen CLIP. GPU memory denotes the peak allocated memory. Table 66 reports the detailed costs.

AuthorsOpenReview ↗

Global response (part 3/3)

Table 6. Computation and memory costs. Cost itemMNISTCelebACIFAR-10Total calibration steps533Metadata reading / image encoding (s)0.3090.77025.019One calibration parameter update (s)606.494127.351223.81HVP time(s)478.393367.491018.81Conditional gradients on Dcal (s)88.80638.79168.07Peak calibration GPU memory (GiB)1.7423.8441.999Responses on Dstat (s)86.41635.4356.82Module-partition search (s)292.5216.94225.70Scoring all candidates (s)555.52770.73375.96Final selection (ms)28.3016.3534.26\begin{array}{lrrr}\hline \text{Cost item} & \text{MNIST} & \text{CelebA} & \text{CIFAR-10} \\\\ \hline \text{Total calibration steps} & 5 & 3 & 3 \\\\ \text{Metadata reading / image encoding (s)} & 0.309 & 0.770 & 25.019 \\\\ \text{One calibration parameter update (s)} & 606.49 & 4127.35 & 1223.81 \\\\ \quad\text{HVP time(s)} & 478.39 & 3367.49 & 1018.81 \\\\ \text{Conditional gradients on }D_{\mathrm{cal}}\text{ (s)} & 88.80 & 638.79 & 168.07 \\\\ \text{Peak calibration GPU memory (GiB)} & 1.742 & 3.844 & 1.999 \\\\ \text{Responses on }D_{\mathrm{stat}}\text{ (s)} & 86.41 & 635.43 & 56.82 \\\\ \text{Module-partition search (s)} & 292.52 & 16.94 & 225.70 \\\\ \text{Scoring all candidates (s)} & 555.52 & 770.73 & 375.96 \\\\ \text{Final selection (ms)} & 28.30 & 16.35 & 34.26 \\\\ \hline\end{array}

Calibration communication per client per step: Table 77 reports the communication cost of each client in one calibration parameter update. The main communication costs come from conditional gradients and the vectors aq,jℓ,(t)\boldsymbol{a}_{q,j}^{\ell,(t)} sent from the server to clients direction vectors, rather than a full Hessian.

Table 7. Calibration communication per client per step. Payload itemMNISTCelebACIFAR-10Gradient upload (GiB)1.1653.3311.082Download of aq,jℓ,(t) (GiB)1.1653.3311.082HVP upload (MiB)42.62142.63942.625Counts / losses upload (bytes)4481280416Total payload (GiB)2.3726.7042.206\begin{array}{lrrr}\hline \text{Payload item} & \text{MNIST} & \text{CelebA} & \text{CIFAR-10} \\\\ \hline \text{Gradient upload (GiB)} & 1.165 & 3.331 & 1.082 \\\\ \text{Download of }\boldsymbol{a}_{q,j}^{\ell,(t)}\text{ (GiB)} & 1.165 & 3.331 & 1.082 \\\\ \text{HVP upload (MiB)} & 42.621 & 42.639 & 42.625 \\\\ \text{Counts / losses upload (bytes)} & 448 & 1280 & 416 \\\\ \text{Total payload (GiB)} & 2.372 & 6.704 & 2.206 \\\\ \hline\end{array}

Evidence of participation for resource-constrained clients: In Table 88, we add 55 resource-constrained clients to each of the three datasets and conduct each test 3 times, resulting in a total of 45 tests. The maximal GPU memory limits are 220220, 216216, and 238238 MiB respectively. The peak training memory usages of the MDMP sub-models are 182.23182.23, 196.22196.22, and 214.18214.18 MiB respectively, all below these limits.Under the same training conditions, the complete global model failed to complete the training 4545 times due to out-of-memory (OOM). However, the MDMP sub-model completed local training and testing 4545 times, indicating that MDMP enables these resource-constrained clients that were previously unable to train the global model to participate.

Table 8. Resource-constrained participation. ItemMNISTCelebACIFAR-10Batch size163216Maximal GPU memory (MiB)220216238Peak training memory (MiB)182.23196.22214.18Full-model OOM15/1515/1515/15MDMP completed15/1515/1515/15Mean accuracy drop (pp)3.1420.1863.750\begin{array}{lrrr}\text{Item} & \text{MNIST} & \text{CelebA} & \text{CIFAR-10} \\\\ \hline \text{Batch size} & 16 & 32 & 16 \\\\ \hline \text{Maximal GPU memory (MiB)} & 220 & 216 & 238 \\\\ \hline \text{Peak training memory (MiB)} & 182.23 & 196.22 & 214.18 \\\\ \hline \text{Full-model OOM} & 15/15 & 15/15 & 15/15 \\\\ \hline \textbf{MDMP completed} & 15/15 & 15/15 & 15/15 \\\\ \hline \text{Mean accuracy drop (pp)} & 3.142 & 0.186 & 3.750\end{array}

Summary of manuscript revisions

\begin{array}{ll}\text{Modified Part} & \text{Description} \\\\ \hline \text{Section 2.2} & \text{Added brief introductions to FIARSE, FedSelect, and SubFLOT.} \\\\ \hline \text{Appendix I.9, Table 7} & \text{Added module-count sensitivity results on MNIST and CIFAR-10} \\\\ & \text{under 95\\% and 90\\% MAC budgets.} \\\\ \hline \text{Appendix J} & \text{Added an overview of the MDMP technical framework.}\end{array}

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.